Oct 30 21:22:14.498906 ip-10-0-134-53 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:22:14.898622 ip-10-0-134-53 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:14.898622 ip-10-0-134-53 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:22:14.898622 ip-10-0-134-53 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:14.898622 ip-10-0-134-53 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:14.902867 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902780 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.902867 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902866 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.902867 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902870 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902874 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902878 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902880 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902883 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902886 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902888 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902891 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902895 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902898 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902900 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902903 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902906 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902908 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902918 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902921 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902923 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902926 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902928 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902931 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.902960 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902933 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902936 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902938 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902941 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902943 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902946 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902948 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902951 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902953 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902956 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902964 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902967 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902969 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902972 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902975 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902983 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902986 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.903420 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902988 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902992 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902996 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.902999 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903001 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903004 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903009 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903011 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903013 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903016 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903019 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903022 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903024 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903026 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903030 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903033 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903035 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903038 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.903922 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903041 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903043 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903048 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903051 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903053 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903055 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903058 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903060 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903063 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903065 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903068 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903075 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903078 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903081 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903083 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903086 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903088 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903090 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903093 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.904374 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903095 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903099 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903103 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903105 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903108 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903111 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903113 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903115 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903118 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903121 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903124 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903127 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903129 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903133 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903136 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903138 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903140 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903143 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903145 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903148 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.904851 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903150 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903152 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903155 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903157 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903160 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.903169 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904873 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904882 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904885 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904888 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904891 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904893 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904895 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904899 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904901 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904904 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904906 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904909 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904911 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904914 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.905334 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904917 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904919 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904922 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904924 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904926 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904929 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904932 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904934 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904937 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904940 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904945 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904948 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904951 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904954 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904957 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904960 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904963 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904966 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904974 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.905826 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904977 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904979 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904982 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904985 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904987 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904989 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904992 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904995 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.904997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905000 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905003 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905005 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905008 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905011 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905013 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905016 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905018 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905020 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905023 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905026 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.906276 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905029 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905031 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905034 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905036 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905039 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905041 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905043 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905049 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905051 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905053 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905056 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905065 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905068 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905070 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905073 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905076 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905078 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905080 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905083 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.906792 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905086 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905088 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905091 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905093 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905098 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905101 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905104 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905106 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905108 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905111 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905114 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905116 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905119 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905121 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905123 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905126 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905128 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905130 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905133 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905135 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.907272 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905138 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905140 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905142 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905145 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905147 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905157 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905161 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905165 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905171 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905174 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905177 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905180 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905766 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905772 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905775 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905780 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905783 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905786 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.907773 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905788 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905791 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905794 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905796 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905799 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905802 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905804 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905808 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905812 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905816 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905819 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905822 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905824 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905827 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905830 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905833 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905836 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905847 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.908322 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905850 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905852 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905855 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905857 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905860 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905862 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905867 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905872 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905875 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905878 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905880 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905882 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905885 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905887 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905889 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905892 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905894 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905897 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.908793 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905899 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905901 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905904 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905906 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905908 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905911 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905913 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905915 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905918 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905920 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905922 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905925 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905933 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905936 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905938 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905941 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905946 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905949 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.909270 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905951 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905954 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905956 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905959 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905961 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905966 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905969 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905971 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905973 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905976 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905978 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905981 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905983 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905987 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905990 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905992 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905995 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.905997 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.909719 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906000 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906002 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906005 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906007 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906009 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906012 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906014 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906022 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906024 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906028 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906030 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906034 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906037 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906039 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906041 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906044 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906046 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906049 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906051 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906054 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.910160 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906056 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906058 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.906061 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907026 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907036 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907043 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907048 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907070 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907075 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907080 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907084 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907087 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907090 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907094 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907097 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907101 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907103 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907106 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907109 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907112 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907115 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907120 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907123 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907127 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:22:14.910663 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907130 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907133 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907137 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907140 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907143 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907147 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907150 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907153 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907155 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907159 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907161 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907166 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907169 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907172 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907174 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907178 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907181 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907186 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907189 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907191 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907194 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907197 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907202 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907205 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907208 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:22:14.911206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907211 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907214 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907217 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907219 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907223 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907226 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907229 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907232 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907236 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907239 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907242 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907247 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907250 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907253 2579 flags.go:64] FLAG: --help="false" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907256 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907259 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907262 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907265 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907268 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907271 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907274 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907277 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907281 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:22:14.911841 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907284 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907287 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907290 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907292 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907295 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907298 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907301 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907304 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907306 2579 flags.go:64] FLAG: --lock-file="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907309 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907312 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907315 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907321 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907324 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907328 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907330 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907333 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907336 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907339 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907342 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907348 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907351 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907355 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907358 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907361 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:22:14.912397 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907364 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907366 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907369 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907372 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907375 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907383 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907386 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907389 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907392 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907395 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907398 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907401 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907403 2579 flags.go:64] FLAG: --port="10250" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907406 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907410 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0472769f3faf3c528" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907413 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907416 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907419 2579 flags.go:64] FLAG: --register-node="true" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907422 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907426 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907429 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907432 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907435 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907438 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907442 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907445 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:22:14.913008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907448 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907452 2579 flags.go:64] FLAG: --runonce="false" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907455 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907457 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907460 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907463 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907466 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907469 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907472 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907476 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907479 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907482 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907485 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907488 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907499 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907503 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907506 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907512 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907515 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907518 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907521 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907524 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907528 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907531 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907533 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:22:14.913623 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907536 2579 flags.go:64] FLAG: --v="2" Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907544 2579 flags.go:64] FLAG: --version="false" Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907549 2579 flags.go:64] FLAG: --vmodule="" Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907553 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.907556 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907720 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907724 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907728 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907733 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907735 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907738 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907741 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907744 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907746 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907749 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907751 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907755 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907758 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907760 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907763 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907766 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.914241 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907771 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907773 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907776 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907781 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907784 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907787 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907790 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907792 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907795 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907797 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907800 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907802 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907805 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907808 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907814 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907816 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.914749 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907820 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907822 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907824 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907827 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907830 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907832 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907834 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907837 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907841 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907843 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907846 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907848 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907851 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907853 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907856 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907859 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907861 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907864 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907867 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.915196 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907869 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907872 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907874 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907877 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907879 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907882 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907884 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907887 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907889 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907893 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907897 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907900 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907903 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907907 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907910 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907912 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907915 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907917 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907920 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.915701 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907922 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907925 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907929 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907932 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907934 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907937 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907939 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907942 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907944 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907947 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907949 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907952 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907955 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907957 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907960 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907962 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907964 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907967 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907969 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907972 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.916168 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907974 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907977 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907979 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907981 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907984 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907987 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907991 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907993 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907996 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.907998 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908001 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908005 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908009 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.908036 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.916650 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908079 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908082 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908085 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908087 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908090 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908093 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908095 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908098 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908100 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908103 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908105 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908108 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908111 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908113 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908116 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908118 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908121 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908123 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908126 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.917004 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908128 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908131 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908133 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908136 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908140 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908143 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908146 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908148 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908151 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908153 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908156 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908158 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908161 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908165 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908169 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908172 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908175 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908177 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908180 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.917446 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908182 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908185 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908188 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908190 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908192 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908195 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908199 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908202 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908204 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908207 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908209 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908211 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908214 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908217 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908219 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908222 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908224 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908233 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908236 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.917901 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908238 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908241 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908243 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908246 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908248 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908251 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908255 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908257 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908260 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908263 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908265 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908268 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908271 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908273 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908276 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908278 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908283 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908286 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908289 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908292 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.918376 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908294 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908297 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908299 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908302 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908304 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908306 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908309 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908311 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908314 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908318 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908320 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908323 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908326 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908330 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908333 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908335 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908338 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908340 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908345 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908347 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.918864 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908350 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908352 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908355 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908358 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908360 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908363 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.908365 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.915430 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.915447 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.916007 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.916014 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916075 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916081 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916084 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916087 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916090 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916094 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916098 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916101 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.919335 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916104 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916107 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916110 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916113 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916115 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916118 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916120 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916123 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916125 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916128 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916131 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916133 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916136 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916138 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916141 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916143 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916146 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916148 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916150 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916153 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.919817 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916156 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916160 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916162 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916165 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916167 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916171 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916173 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916176 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916179 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916184 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916187 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916189 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916192 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916194 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916196 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916201 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916204 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916206 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.920274 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916211 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916214 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916217 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916220 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916222 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916225 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916227 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916233 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916235 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916238 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916241 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916245 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916250 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916253 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916256 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916258 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916261 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.920798 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916263 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916266 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916268 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916271 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916276 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916278 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916281 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916283 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916286 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916288 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916291 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916293 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916295 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916298 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916301 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916304 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916306 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916308 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.921245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916311 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916314 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916317 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916319 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916321 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916324 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916326 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916329 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916331 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916334 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916337 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916339 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916341 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916344 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916346 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916351 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916353 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916355 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.921808 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916358 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.916385 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916523 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916528 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916531 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916533 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916536 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916538 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916541 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916544 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916546 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916549 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916551 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916554 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916556 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916559 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.922245 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916561 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916564 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916567 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916569 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916572 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916574 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916576 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916580 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916583 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916585 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916588 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916590 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916596 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916619 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916623 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916626 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916629 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916631 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.922643 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916634 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916637 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916639 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916642 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916645 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916648 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916650 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916653 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916656 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916658 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916661 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916664 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916666 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916669 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916671 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916674 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916676 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916679 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916681 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.923124 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916684 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916686 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916690 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916692 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916695 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916697 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916700 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916702 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916704 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916707 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916709 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916712 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916714 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916717 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916719 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916721 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916725 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916729 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916732 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.923929 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916735 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916737 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916740 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916742 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916745 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916747 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916749 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916752 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916754 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916757 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916760 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916762 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916764 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916767 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916769 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916771 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916774 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916777 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916782 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.924657 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916784 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916786 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916789 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916792 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916794 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916796 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916799 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916801 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916804 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916806 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916809 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916811 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916814 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916816 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916819 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.916841 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:14.925133 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916875 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916879 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916883 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916886 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916889 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916891 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916894 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916896 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916899 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916901 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916903 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916906 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.916908 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917291 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917295 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917297 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917300 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917302 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917305 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:14.925520 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917308 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917310 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917313 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917315 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917318 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917320 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917323 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917325 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917328 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917330 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917332 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917335 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917337 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917340 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917342 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917346 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917350 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917352 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917355 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:14.926064 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917358 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917360 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917363 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917365 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917367 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917370 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917375 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917378 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917381 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917383 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917386 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917388 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917393 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917395 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917398 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917400 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917405 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917408 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:14.926712 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917410 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917413 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917415 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917418 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917421 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917423 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917426 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917429 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917431 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917434 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917436 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917438 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917441 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917443 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917446 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917448 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917451 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917454 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917459 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:14.927188 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917461 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917464 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917467 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917469 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917472 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917474 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917479 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917481 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917484 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917487 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917489 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917492 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917494 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917497 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917499 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917502 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917504 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917507 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917510 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917512 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:14.927665 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917515 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917521 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917525 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917528 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917531 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917534 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917537 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:14.917540 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.917599 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.919623 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.920601 2579 server.go:1014] "Starting client certificate rotation" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.920717 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.920752 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.923720 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:14.928151 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.925988 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:14.934202 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.934184 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:22:14.937381 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.937364 2579 log.go:25] "Validated CRI v1 image API" Oct 30 21:22:14.938264 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.938235 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:22:14.940517 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.940497 2579 fs.go:135] Filesystem UUIDs: map[5ece9ed1-1ef7-4ec3-9c2f-c330abd1f23c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7236199-0d43-45f8-a3d5-785967c5162c:/dev/nvme0n1p3] Oct 30 21:22:14.940569 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.940517 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:22:14.947216 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.947096 2579 manager.go:217] Machine: {Timestamp:2026-10-30 21:22:14.945600117 +0000 UTC m=+0.328359796 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097397 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9c9728a077a88b828b933d4cf503 SystemUUID:ec2b9c97-28a0-77a8-8b82-8b933d4cf503 BootID:c044d4ec-51d6-44bc-bbe2-11214d5ea0ed Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:68:7b:53:c0:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:68:7b:53:c0:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:21:23:03:ad:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:22:14.947803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.947792 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:22:14.947896 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.947883 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:22:14.951026 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.950999 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:22:14.951165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.951030 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:22:14.951210 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.951167 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:22:14.951210 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.951176 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:22:14.951265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.951213 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:14.951265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.951231 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:14.952083 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.952073 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:22:14.955926 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.955914 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:22:14.956052 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956043 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:22:14.956752 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956742 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:22:14.956792 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956759 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:22:14.956792 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956772 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:22:14.956792 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956783 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:22:14.956792 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.956791 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:22:14.957168 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.957156 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:14.959792 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.959764 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:22:14.965722 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.965705 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:14.967521 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967502 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:22:14.967577 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967538 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:22:14.967742 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967732 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:22:14.967799 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967749 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:22:14.967799 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967755 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:22:14.967799 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967761 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:22:14.967799 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967766 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:22:14.967799 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.967772 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:22:14.968411 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968402 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:22:14.968441 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968412 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:22:14.968441 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968420 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:22:14.968441 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968426 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:22:14.968441 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968435 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:22:14.968641 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.968443 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:22:14.969783 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.969774 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:22:14.969866 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.969784 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:22:14.973701 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.973689 2579 server.go:1263] "Started kubelet" Oct 30 21:22:14.973787 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.973746 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:22:14.973877 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.973831 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:22:14.973926 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.973900 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:22:14.974576 ip-10-0-134-53 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:22:14.975895 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.975879 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:22:14.976682 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.976667 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:22:14.979881 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:14.979863 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:22:14.980456 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.980437 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:22:14.981112 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981096 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:22:14.981227 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:14.981208 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:14.981833 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981813 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:22:14.981911 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981836 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:22:14.981911 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981839 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:22:14.981911 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981908 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:22:14.982260 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.981916 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:22:14.982260 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.982018 2579 factory.go:55] Registering systemd factory Oct 30 21:22:14.982260 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.982124 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:22:14.983354 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.983337 2579 factory.go:153] Registering CRI-O factory Oct 30 21:22:14.983354 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.983353 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 21:22:14.983498 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.983393 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:22:14.983498 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.983417 2579 factory.go:103] Registering Raw factory Oct 30 21:22:14.983498 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.983429 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 21:22:14.984263 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.984248 2579 manager.go:319] Starting recovery of all containers Oct 30 21:22:14.997178 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:14.997013 2579 manager.go:324] Recovery completed Oct 30 21:22:15.001310 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.001295 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.003057 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003041 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.003106 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003072 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.003106 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003083 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.003635 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003620 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:22:15.003635 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003633 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:22:15.003717 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003622 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.003717 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.003650 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:22:15.004290 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.004276 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.004341 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.004301 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.004341 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.004311 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.005812 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.005798 2579 policy_none.go:50] "Start" Oct 30 21:22:15.005867 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.005819 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:22:15.005867 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.005834 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:22:15.007997 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.007983 2579 policy_none.go:44] "Start" Oct 30 21:22:15.057308 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.057263 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:22:15.061260 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061218 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 21:22:15.061260 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.061256 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:22:15.061260 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061265 2579 server.go:87] "Starting device plugin registration server" Oct 30 21:22:15.061582 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061567 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:22:15.061661 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061583 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:22:15.061769 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061752 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:22:15.061885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061867 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:22:15.061885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.061876 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:22:15.062475 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.062454 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:22:15.062546 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.062511 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:15.120769 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.120733 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:22:15.122859 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.122124 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:22:15.122859 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.122144 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:22:15.122859 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.122163 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:22:15.122859 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.122195 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:22:15.162480 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.162381 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.163592 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.163575 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.163682 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.163627 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.163682 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.163643 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.163682 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.163672 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.175220 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.175175 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-53.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:22:15.175349 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.175283 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-53.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:22:15.175910 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.174955 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-53.us-west-1.compute.internal.18e36b8e2af28b2f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-53.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:22:14.973655855 +0000 UTC m=+0.356415532,LastTimestamp:2026-10-30 21:22:14.973655855 +0000 UTC m=+0.356415532,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-53.us-west-1.compute.internal,}" Oct 30 21:22:15.202773 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.202747 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:22:15.202885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.202795 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6tkhz" Oct 30 21:22:15.222856 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.222824 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal"] Oct 30 21:22:15.222932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.222897 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.223922 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.223906 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.223994 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.223935 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.223994 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.223948 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.226181 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.226169 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.226861 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.226833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.226985 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.226878 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.227048 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.226993 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.227048 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.227034 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.227128 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.227055 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.227811 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.227797 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.227867 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.227823 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.227867 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.227835 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.229161 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.229147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.229213 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.229172 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:15.229912 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.229894 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:15.230023 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.229923 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:15.230023 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.229933 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:15.260181 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.260157 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.262894 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.262877 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.283048 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.283020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.283144 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.283050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.283144 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.283071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.284518 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.284502 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:15.284576 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.284548 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:15.286578 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.286564 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:15.383952 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.383915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.383952 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.383954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.384195 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.384021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.384195 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.384035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.384195 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.384080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.384195 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.384133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.384391 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.384376 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:15.385764 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.385749 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.385832 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.385776 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-53.us-west-1.compute.internal\": node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:15.481729 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.481656 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.542501 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.542478 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:15.542588 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.542574 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.563455 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.563434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.566599 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.566581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:15.600486 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.600466 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:15.920774 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.920683 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:22:15.921970 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.920852 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:22:15.921970 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.920877 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:22:15.921970 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.920924 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:22:15.921970 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:15.920925 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af1d03f21ae4249649589735f9d6a3af-3cfe10bb520c60ba.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-53.us-west-1.compute.internal.18e36b8e2cb39e3d\": read tcp 10.0.134.53:36788->3.131.73.99:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-53.us-west-1.compute.internal.18e36b8e2cb39e3d default 5065 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-53.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-53.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-53.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:22:15 +0000 UTC,LastTimestamp:2026-10-30 21:22:15.163646795 +0000 UTC m=+0.546406473,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-53.us-west-1.compute.internal,}" Oct 30 21:22:15.957801 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.957776 2579 apiserver.go:52] "Watching apiserver" Oct 30 21:22:15.981402 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:15.981378 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:22:16.207536 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.207418 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:17:15 +0000 UTC" deadline="2028-06-24 04:39:09.226422818 +0000 UTC" Oct 30 21:22:16.207536 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.207476 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14455h16m53.018951851s" Oct 30 21:22:16.234432 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.234398 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:22:16.234751 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.234733 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-25spq","kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-fsslh","openshift-image-registry/node-ca-pgkp4","openshift-dns/node-resolver-czp4b","openshift-multus/multus-6clh6","openshift-network-diagnostics/network-check-target-8r9s9","openshift-network-operator/iptables-alerter-cx46g","kube-system/global-pull-secret-syncer-q7xnk","openshift-multus/multus-additional-cni-plugins-2jlkf","openshift-ovn-kubernetes/ovnkube-node-vh6nl","kube-system/konnectivity-agent-pq7d9"] Oct 30 21:22:16.237173 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.237154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.239183 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.239159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.239311 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.239259 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:16.241242 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.241229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.243264 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.243249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.245333 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.245318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.247375 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.247363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.249369 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.249356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:16.249432 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.249415 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:16.251528 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.251507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.253930 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.253681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.253930 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.253748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:16.255946 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.255927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.258123 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.258108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.260383 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.260364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.282879 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.282857 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:22:16.289348 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.289447 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.289447 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqg6t\" (UniqueName: \"kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.289533 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.289533 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.289617 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.289657 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.289704 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.289704 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.289803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.289803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.289885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.289885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.289885 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.290011 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290011 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.289957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290077 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.290077 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.290141 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzlp\" (UniqueName: \"kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.290141 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290201 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290201 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.290201 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290319 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290356 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290335 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290336 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290471 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqzlq\"" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpq5\" (UniqueName: \"kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.290639 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhvt\" (UniqueName: \"kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gl4\" (UniqueName: \"kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.290958 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.290957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57q6\" (UniqueName: \"kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.291247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvsv\" (UniqueName: \"kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z2q8\" (UniqueName: \"kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:16.291696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj28h\" (UniqueName: \"kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.291999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.292017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.292165 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.292037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.293727 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.293710 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.293871 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.293855 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.294076 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.294059 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbjr2\"" Oct 30 21:22:16.295759 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.295744 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf4nn\"" Oct 30 21:22:16.295826 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.295786 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:22:16.295826 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.295790 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.295894 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.295824 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.299872 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.299593 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gkfxf\"" Oct 30 21:22:16.351864 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.351828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.352007 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.351887 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.352007 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.351892 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.352007 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.351887 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.352007 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.351943 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:22:16.385518 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.385476 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:22:16.385651 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.385534 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx2jw" Oct 30 21:22:16.391436 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391409 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:22:16.391649 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391529 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:22:16.391649 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391584 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391783 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391802 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391820 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ntkmx\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391845 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391871 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391804 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391897 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391928 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391940 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391900 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391930 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.391973 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bcb\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392014 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vm9q9\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392040 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:22:16.392056 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392041 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gmgqb\"" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392088 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lclr\"" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392353 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj28h\" (UniqueName: \"kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.392520 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392657 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392707 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg6t\" (UniqueName: \"kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392746 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393063 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.392987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393101 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393130 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393355 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.393975 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzlp\" (UniqueName: \"kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393824 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393852 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.394761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpq5\" (UniqueName: \"kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.393987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhvt\" (UniqueName: \"kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gl4\" (UniqueName: \"kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394494 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.394640 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57q6\" (UniqueName: \"kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.395797 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.394993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvsv\" (UniqueName: \"kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2q8\" (UniqueName: \"kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395257 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395877 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395932 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.395978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396038 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.396932 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396450 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.396502 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:16.896468842 +0000 UTC m=+2.279228526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396509 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.396722 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.396769 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:16.896755032 +0000 UTC m=+2.279514703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396924 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396955 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.396987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397163 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397211 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397214 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397323 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397367 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.397960 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.398803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.398803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.398803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.398803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.397856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.454787 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.454750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg6t\" (UniqueName: \"kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.454953 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.454799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzlp\" (UniqueName: \"kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.455019 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.454963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj28h\" (UniqueName: \"kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.455214 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.455189 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpq5\" (UniqueName: \"kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.456166 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.456137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhvt\" (UniqueName: \"kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.456323 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.456308 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvsv\" (UniqueName: \"kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.456968 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.456954 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:16.457015 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.456971 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:16.457015 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.456981 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:16.457460 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.457441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2q8\" (UniqueName: \"kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.457689 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.457649 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:16.957631038 +0000 UTC m=+2.340390702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:16.458977 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.458953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57q6\" (UniqueName: \"kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.459553 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.459538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gl4\" (UniqueName: \"kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.578149 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.578114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:16.595629 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.595567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:16.604788 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.604759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:16.617642 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.617602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:16.628789 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.628761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6clh6" Oct 30 21:22:16.647471 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.647435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:16.670713 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.670669 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6010805d267929c8af4d6a80598dea.slice/crio-680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1 WatchSource:0}: Error finding container 680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1: Status 404 returned error can't find the container with id 680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1 Oct 30 21:22:16.671381 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.671352 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ab472e_ac86_4e2a_9939_148e357f191b.slice/crio-af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9 WatchSource:0}: Error finding container af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9: Status 404 returned error can't find the container with id af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9 Oct 30 21:22:16.679952 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.679926 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:22:16.682670 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.682644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:16.686255 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.686229 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d17c7cfc239df4f9cffce0fad7645b.slice/crio-b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e WatchSource:0}: Error finding container b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e: Status 404 returned error can't find the container with id b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e Oct 30 21:22:16.692581 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.692553 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f515ccd_ec8c_4407_8a96_72f3cf960b05.slice/crio-2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb WatchSource:0}: Error finding container 2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb: Status 404 returned error can't find the container with id 2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb Oct 30 21:22:16.694740 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.694714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:16.703863 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.703833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:16.704949 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.704927 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e468f3_d3a6_4f2d_8964_f18c273b0db3.slice/crio-d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121 WatchSource:0}: Error finding container d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121: Status 404 returned error can't find the container with id d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121 Oct 30 21:22:16.712982 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:16.712954 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36343aa1_c487_43ea_9f03_9d639fad9893.slice/crio-d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7 WatchSource:0}: Error finding container d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7: Status 404 returned error can't find the container with id d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7 Oct 30 21:22:16.898713 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.898673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:16.898897 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.898737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:16.898897 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.898852 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:16.898897 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.898854 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:16.899003 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.898913 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:17.898898422 +0000 UTC m=+3.281658088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:16.899003 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.898943 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:17.898929246 +0000 UTC m=+3.281688914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:16.999605 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:16.999501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:17.000199 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.999703 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:17.000199 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.999718 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:17.000199 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.999729 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:17.000199 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:16.999782 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:17.99976878 +0000 UTC m=+3.382528447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:17.126679 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.126595 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq7d9" event={"ID":"85e468f3-d3a6-4f2d-8964-f18c273b0db3","Type":"ContainerStarted","Data":"d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121"} Oct 30 21:22:17.127812 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.127777 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerStarted","Data":"2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb"} Oct 30 21:22:17.128927 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.128903 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerStarted","Data":"b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e"} Oct 30 21:22:17.130068 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.130049 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cx46g" event={"ID":"68ab472e-ac86-4e2a-9939-148e357f191b","Type":"ContainerStarted","Data":"af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9"} Oct 30 21:22:17.131097 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.131074 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"ad6010805d267929c8af4d6a80598dea","Type":"ContainerStarted","Data":"680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1"} Oct 30 21:22:17.132064 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.132037 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7"} Oct 30 21:22:17.176509 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.176479 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:17.214061 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.214028 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:17.388122 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.388003 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:16 +0000 UTC" deadline="2028-07-21 22:46:04.681628094 +0000 UTC" Oct 30 21:22:17.388122 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.388061 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15121h23m47.293570515s" Oct 30 21:22:17.426744 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.426704 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:17.444634 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:17.444594 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edb46bb_cfe0_4220_8843_c30652a67b40.slice/crio-9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57 WatchSource:0}: Error finding container 9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57: Status 404 returned error can't find the container with id 9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57 Oct 30 21:22:17.472324 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:17.472283 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975bc6cc_1f63_4ae5_ac26_771225c652b4.slice/crio-23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf WatchSource:0}: Error finding container 23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf: Status 404 returned error can't find the container with id 23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf Oct 30 21:22:17.482886 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:17.482860 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abb6ccb_d1fa_45e4_88d7_84ed3a7a0eec.slice/crio-e2536e0a5aec0db9fb8c1ff473d40798f7a766e739f65bf8f009c68248cb27b3 WatchSource:0}: Error finding container e2536e0a5aec0db9fb8c1ff473d40798f7a766e739f65bf8f009c68248cb27b3: Status 404 returned error can't find the container with id e2536e0a5aec0db9fb8c1ff473d40798f7a766e739f65bf8f009c68248cb27b3 Oct 30 21:22:17.531770 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:17.531737 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a817f0e_4473_409d_9638_459e8b0a066f.slice/crio-c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870 WatchSource:0}: Error finding container c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870: Status 404 returned error can't find the container with id c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870 Oct 30 21:22:17.554430 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:17.554395 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded19e99d_4341_4066_bb33_4fd4361b795a.slice/crio-bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42 WatchSource:0}: Error finding container bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42: Status 404 returned error can't find the container with id bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42 Oct 30 21:22:17.905496 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.905454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:17.905728 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:17.905546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:17.905728 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:17.905714 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:17.905855 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:17.905781 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:19.905761863 +0000 UTC m=+5.288521547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:17.905923 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:17.905875 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:17.905923 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:17.905913 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:19.905899359 +0000 UTC m=+5.288659044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:18.006183 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.006083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:18.006754 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.006426 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:18.006754 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.006514 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:18.006754 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.006532 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:18.006754 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.006583 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:20.006568802 +0000 UTC m=+5.389328483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:18.122633 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.122587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:18.122840 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.122758 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:18.122840 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.122784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:18.122953 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.122918 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:18.123008 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.122952 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:18.123061 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:18.123023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:18.148581 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.148342 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fsslh" event={"ID":"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec","Type":"ContainerStarted","Data":"e2536e0a5aec0db9fb8c1ff473d40798f7a766e739f65bf8f009c68248cb27b3"} Oct 30 21:22:18.153617 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.153497 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" event={"ID":"7edb46bb-cfe0-4220-8843-c30652a67b40","Type":"ContainerStarted","Data":"9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57"} Oct 30 21:22:18.155572 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.155501 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6clh6" event={"ID":"ed19e99d-4341-4066-bb33-4fd4361b795a","Type":"ContainerStarted","Data":"bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42"} Oct 30 21:22:18.165022 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.164993 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czp4b" event={"ID":"9a817f0e-4473-409d-9638-459e8b0a066f","Type":"ContainerStarted","Data":"c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870"} Oct 30 21:22:18.176191 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.176160 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgkp4" event={"ID":"975bc6cc-1f63-4ae5-ac26-771225c652b4","Type":"ContainerStarted","Data":"23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf"} Oct 30 21:22:18.390178 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.389060 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:16 +0000 UTC" deadline="2028-07-17 02:43:40.191508974 +0000 UTC" Oct 30 21:22:18.390178 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:18.389104 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15005h21m21.802410481s" Oct 30 21:22:19.137572 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:19.137239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:19.138073 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:19.137645 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:19.925100 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:19.925055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:19.925304 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:19.925148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:19.925304 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:19.925292 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:19.925449 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:19.925366 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:23.925347921 +0000 UTC m=+9.308107611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:19.925523 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:19.925457 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:19.925523 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:19.925499 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:23.925483962 +0000 UTC m=+9.308243634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:20.025865 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:20.025725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:20.026032 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.025958 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:20.026032 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.025981 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:20.026032 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.025995 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:20.026264 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.026055 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:24.026037254 +0000 UTC m=+9.408796946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:20.123378 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:20.122991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:20.123378 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.123138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:20.123378 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:20.123233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:20.123378 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:20.123333 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:21.125816 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:21.125767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:21.126333 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:21.125912 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:21.208726 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:21.208331 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="9dda932121cbf9c2005b8535018741af122300d114cc23033f914fcfce8a74b4" exitCode=0 Oct 30 21:22:21.208726 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:21.208398 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"9dda932121cbf9c2005b8535018741af122300d114cc23033f914fcfce8a74b4"} Oct 30 21:22:22.122783 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:22.122404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:22.122783 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:22.122522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:22.122783 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:22.122587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:22.122783 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:22.122687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:22.212023 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:22.211987 2579 generic.go:356] "Generic (PLEG): container finished" podID="d7d17c7cfc239df4f9cffce0fad7645b" containerID="b972b111fe259520d0c42a525702f04d82147e347a5ba4dfbd24bf13c2dd0ae0" exitCode=0 Oct 30 21:22:22.212429 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:22.212064 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerDied","Data":"b972b111fe259520d0c42a525702f04d82147e347a5ba4dfbd24bf13c2dd0ae0"} Oct 30 21:22:22.227883 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:22.227347 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pq7d9" event={"ID":"85e468f3-d3a6-4f2d-8964-f18c273b0db3","Type":"ContainerStarted","Data":"da4b400b30c228d4ae35226a742c43371288e46b6f36dd6fa45461d3a6d6a35d"} Oct 30 21:22:23.125206 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.125178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:23.125382 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:23.125285 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:23.233294 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.233255 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerStarted","Data":"dd1d0b352519250d02a0a092c165ed87fcc7456bf959056848ee32454a197963"} Oct 30 21:22:23.238866 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.238835 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"ad6010805d267929c8af4d6a80598dea","Type":"ContainerStarted","Data":"f4146dd1f53b4bbea4b0fd7b7d6cd5d7f0b6df36cbf1b198189e2767948f23c0"} Oct 30 21:22:23.350541 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.349054 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pq7d9" podStartSLOduration=3.686081957 podStartE2EDuration="8.349031824s" totalImagesPullingTime="4.662949867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:22.449732641 +0000 UTC m=+7.832492329" watchObservedRunningTime="2026-10-30 21:22:23.349031824 +0000 UTC m=+8.731791512" Oct 30 21:22:23.467126 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.465576 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" podStartSLOduration=8.465556292 podStartE2EDuration="8.465556292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:23.350321636 +0000 UTC m=+8.733081325" watchObservedRunningTime="2026-10-30 21:22:23.465556292 +0000 UTC m=+8.848315984" Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.960034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:23.960128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:23.960270 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:23.960340 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:31.960321225 +0000 UTC m=+17.343080904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:23.960407 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:23.960495 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:23.960452 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:31.960432416 +0000 UTC m=+17.343192102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:24.060734 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.060693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:24.060907 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.060872 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:24.060907 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.060893 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:24.061013 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.060905 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:24.061065 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.061049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:32.061035938 +0000 UTC m=+17.443795603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:24.123129 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.123097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:24.123295 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.123235 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:24.123295 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.123292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:24.123412 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:24.123344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:24.245300 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.245198 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pgkp4" event={"ID":"975bc6cc-1f63-4ae5-ac26-771225c652b4","Type":"ContainerStarted","Data":"83b52cbb77d3e1a8da60ce065f8e616fd4ece3841e94eaf33c5f959f1f131ef7"} Oct 30 21:22:24.361533 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.361468 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" podStartSLOduration=9.361441185 podStartE2EDuration="9.361441185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:23.467089653 +0000 UTC m=+8.849849343" watchObservedRunningTime="2026-10-30 21:22:24.361441185 +0000 UTC m=+9.744200873" Oct 30 21:22:24.361765 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.361659 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pgkp4" podStartSLOduration=3.377423029 podStartE2EDuration="9.361648393s" totalImagesPullingTime="5.984225364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:24.360752024 +0000 UTC m=+9.743511712" watchObservedRunningTime="2026-10-30 21:22:24.361648393 +0000 UTC m=+9.744408081" Oct 30 21:22:24.575112 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.574183 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:24.575681 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:24.575496 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:25.124466 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:25.124419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:25.124667 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:25.124640 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:25.252346 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:25.252309 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" event={"ID":"7edb46bb-cfe0-4220-8843-c30652a67b40","Type":"ContainerStarted","Data":"02c8e7c5888a58e2d1aa0e69e64a2a97cb3ac604945a09f6a31e75dd86ddf59a"} Oct 30 21:22:26.135402 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.134838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:26.135402 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:26.135190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:26.135402 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.135274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:26.135402 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:26.135345 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:26.260353 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.260238 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerStarted","Data":"555f32ef815b4f31558ee82a5c6fdbc386fb1d608104c1dd66686e282f4e9960"} Oct 30 21:22:26.262649 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.262283 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6clh6" event={"ID":"ed19e99d-4341-4066-bb33-4fd4361b795a","Type":"ContainerStarted","Data":"26724c2a5052c20c5fb814ac384e33ea1e650fa907659d7da350413da887dfab"} Oct 30 21:22:26.504328 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.504215 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6clh6" podStartSLOduration=3.089258751 podStartE2EDuration="11.504196766s" totalImagesPullingTime="8.414938015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:26.503982864 +0000 UTC m=+11.886742552" watchObservedRunningTime="2026-10-30 21:22:26.504196766 +0000 UTC m=+11.886956453" Oct 30 21:22:26.760794 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:26.760764 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:22:27.074480 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.074295 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:22:26.760790411Z","UUID":"ecbf7aa4-0888-436e-910e-109efb14465c","Handler":null,"Name":"","Endpoint":""} Oct 30 21:22:27.079945 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.079915 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:22:27.080115 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.079955 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:22:27.123491 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.123455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:27.123677 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:27.123636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:27.268536 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.268503 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="555f32ef815b4f31558ee82a5c6fdbc386fb1d608104c1dd66686e282f4e9960" exitCode=0 Oct 30 21:22:27.269169 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.268592 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"555f32ef815b4f31558ee82a5c6fdbc386fb1d608104c1dd66686e282f4e9960"} Oct 30 21:22:27.271597 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.271571 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cx46g" event={"ID":"68ab472e-ac86-4e2a-9939-148e357f191b","Type":"ContainerStarted","Data":"18f9d4c283c2626ea112c9d5d9c7eb651df4260af82c52b1dd19024aee01afad"} Oct 30 21:22:27.273781 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.273754 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" event={"ID":"7edb46bb-cfe0-4220-8843-c30652a67b40","Type":"ContainerStarted","Data":"a86d64fc20125b8d1d9b71f3f4f38aa273e32ac4c54ce37322812b3d0ef5b729"} Oct 30 21:22:27.275428 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.275401 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-czp4b" event={"ID":"9a817f0e-4473-409d-9638-459e8b0a066f","Type":"ContainerStarted","Data":"5c8c447c7761209721bbc7653aeba8d709ec4cbeb44bc70f662d9ef661994192"} Oct 30 21:22:27.623673 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.623597 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-czp4b" podStartSLOduration=3.909072849 podStartE2EDuration="12.62358191s" totalImagesPullingTime="8.714509061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:27.623458784 +0000 UTC m=+13.006218472" watchObservedRunningTime="2026-10-30 21:22:27.62358191 +0000 UTC m=+13.006341594" Oct 30 21:22:27.623857 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:27.623790 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cx46g" podStartSLOduration=3.664039655 podStartE2EDuration="12.623780189s" totalImagesPullingTime="8.959740534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:27.506721236 +0000 UTC m=+12.889480923" watchObservedRunningTime="2026-10-30 21:22:27.623780189 +0000 UTC m=+13.006539879" Oct 30 21:22:28.123199 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:28.123121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:28.123361 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:28.123262 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:28.123361 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:28.123342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:28.123464 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:28.123424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:29.123719 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.123440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:29.124264 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:29.123754 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:29.282135 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.282098 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="d9f5587bfd2590c17246a89eebacd544597b33b7425a9a7b0bd15c8b82d9562f" exitCode=0 Oct 30 21:22:29.282314 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.282193 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"d9f5587bfd2590c17246a89eebacd544597b33b7425a9a7b0bd15c8b82d9562f"} Oct 30 21:22:29.288955 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.285855 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fsslh" event={"ID":"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec","Type":"ContainerStarted","Data":"f216e108cbc51c1b622e620304484a8113a6d3615331e0c3b25a98084fb7db74"} Oct 30 21:22:29.290075 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.289576 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" event={"ID":"7edb46bb-cfe0-4220-8843-c30652a67b40","Type":"ContainerStarted","Data":"81ebce352c0dd3a29a2203c4cfa42ec5e6780238b7147faa59a8fe0458ffa4f0"} Oct 30 21:22:29.574189 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.574088 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" podStartSLOduration=3.472506416 podStartE2EDuration="14.574072164s" totalImagesPullingTime="11.101565748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:29.573562256 +0000 UTC m=+14.956321943" watchObservedRunningTime="2026-10-30 21:22:29.574072164 +0000 UTC m=+14.956831850" Oct 30 21:22:29.691947 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:29.691890 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fsslh" podStartSLOduration=3.835462706 podStartE2EDuration="14.69187665s" totalImagesPullingTime="10.856413944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:29.691462765 +0000 UTC m=+15.074222455" watchObservedRunningTime="2026-10-30 21:22:29.69187665 +0000 UTC m=+15.074636336" Oct 30 21:22:30.122576 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:30.122533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:30.122576 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:30.122573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:30.122839 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:30.122694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:30.122943 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:30.122834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:31.122559 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:31.122475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:31.122950 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:31.122636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:31.297713 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:31.297671 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="984f1626f5bbda06d5d555f088c4aaffcfe6d578b44beb04c64c4557e947c7c8" exitCode=0 Oct 30 21:22:31.297713 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:31.297718 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"984f1626f5bbda06d5d555f088c4aaffcfe6d578b44beb04c64c4557e947c7c8"} Oct 30 21:22:32.031952 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:32.031909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:32.032166 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:32.031961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:32.032166 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.032074 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:32.032166 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.032094 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:32.032166 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.032134 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:48.032121717 +0000 UTC m=+33.414881402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:22:32.032166 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.032167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret podName:f3f8276e-9414-4129-8332-22be9c980723 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:48.032146033 +0000 UTC m=+33.414905701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret") pod "global-pull-secret-syncer-q7xnk" (UID: "f3f8276e-9414-4129-8332-22be9c980723") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:22:32.122927 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:32.122885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:32.122927 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:32.122916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:32.123426 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.123032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:32.123426 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.123156 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:32.133067 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:32.133037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:32.133221 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.133182 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:22:32.133221 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.133197 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:22:32.133221 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.133205 2579 projected.go:196] Error preparing data for projected volume kube-api-access-wxsxx for pod openshift-network-diagnostics/network-check-target-8r9s9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:32.133340 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:32.133272 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx podName:ab75e151-6ab3-4451-abf4-abb25a2e0463 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:48.133252291 +0000 UTC m=+33.516011979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxsxx" (UniqueName: "kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx") pod "network-check-target-8r9s9" (UID: "ab75e151-6ab3-4451-abf4-abb25a2e0463") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:22:33.126748 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.126525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:33.127154 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:33.126869 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:33.305365 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.305337 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"01af555a26535a91b551734aa393cfe2e634d96391621129ac4f6e97620d38ec"} Oct 30 21:22:33.305485 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.305407 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"9814bde1687cec8fc7ef81abe92e64a8c4bb61f575d24f9d5ec2e2820374be6f"} Oct 30 21:22:33.305485 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.305452 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"0ca32953b387b21973e11beb86e789acd0a45412402c88ae710f4054bb139087"} Oct 30 21:22:33.305485 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.305482 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"f09284ecda038c71375e8250e19d476f69e1891cc62814f728885312bf0a5727"} Oct 30 21:22:33.305588 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:33.305507 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"0e919f59bb6a03a9984b1505f090324a767dfa8df5567b302d0d99150774bbfa"} Oct 30 21:22:34.123239 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:34.123153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:34.123239 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:34.123188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:34.123409 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:34.123281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:34.123409 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:34.123361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:34.310486 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:34.310451 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"12337475af4dff0be3d09bdad8039953e41e4730acf3371aa9ada1d30f1d472e"} Oct 30 21:22:34.313269 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:34.313241 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="facfdb94354216ec46545e54a78131f669e5762434dec67b85af3ea9c4c542dd" exitCode=0 Oct 30 21:22:34.313347 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:34.313300 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"facfdb94354216ec46545e54a78131f669e5762434dec67b85af3ea9c4c542dd"} Oct 30 21:22:35.125274 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:35.125245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:35.125443 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:35.125336 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:35.318552 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:35.318517 2579 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="eb78fc763a968f6a75ed2ccc7635ad4923056cd3c87db3efa3002482000f95e4" exitCode=0 Oct 30 21:22:35.319155 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:35.318579 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerDied","Data":"eb78fc763a968f6a75ed2ccc7635ad4923056cd3c87db3efa3002482000f95e4"} Oct 30 21:22:36.122765 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:36.122728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:36.122765 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:36.122751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:36.122941 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:36.122846 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:36.122991 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:36.122973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:36.324378 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:36.324347 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"9a8aedae2b04e18538cac66ec877563cb19800563399cd913665ee0b64f56de3"} Oct 30 21:22:36.327537 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:36.327506 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" event={"ID":"1f515ccd-ec8c-4407-8a96-72f3cf960b05","Type":"ContainerStarted","Data":"4f267122304c16525454d160e9382b3e0ade578a4d95ad907d63cca4c449d457"} Oct 30 21:22:36.453286 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:36.453170 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2jlkf" podStartSLOduration=8.078047247 podStartE2EDuration="21.453155177s" totalImagesPullingTime="13.37510793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:36.452662964 +0000 UTC m=+21.835422656" watchObservedRunningTime="2026-10-30 21:22:36.453155177 +0000 UTC m=+21.835914863" Oct 30 21:22:37.122876 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:37.122850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:37.123010 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:37.122947 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:38.122676 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.122467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:38.123101 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.122512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:38.123101 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:38.122755 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:38.123101 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:38.122834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:38.335187 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.335155 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" event={"ID":"36343aa1-c487-43ea-9f03-9d639fad9893","Type":"ContainerStarted","Data":"0fc0c43a0e1c2a3646c5b264e79b4be8e0e7518b991dd068fa5df68e6166a840"} Oct 30 21:22:38.335443 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.335427 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:38.351899 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.351875 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:38.615692 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:38.615573 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" podStartSLOduration=7.345813441 podStartE2EDuration="23.615559821s" totalImagesPullingTime="16.26974638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:38.615251771 +0000 UTC m=+23.998011457" watchObservedRunningTime="2026-10-30 21:22:38.615559821 +0000 UTC m=+23.998319508" Oct 30 21:22:39.123047 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:39.123012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:39.123432 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:39.123162 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:39.338090 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:39.338060 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:39.338268 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:39.338140 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:39.353457 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:39.353430 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:40.122734 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.122704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:40.122904 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.122743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:40.122904 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:40.122827 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:40.123014 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:40.122948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:40.347584 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.347489 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7xnk"] Oct 30 21:22:40.348018 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.347589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:40.348018 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:40.347699 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:40.350517 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.350490 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r9s9"] Oct 30 21:22:40.350669 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.350579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:40.350736 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:40.350673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:40.351178 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.351156 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-25spq"] Oct 30 21:22:40.351265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:40.351250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:40.351354 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:40.351336 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:42.123132 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:42.123098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:42.123514 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:42.123141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:42.123514 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:42.123194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:42.123514 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:42.123208 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:42.123514 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:42.123272 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:42.123514 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:42.123361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:43.496130 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:43.496103 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:43.496515 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:43.496236 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:22:43.496772 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:43.496755 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:44.122675 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:44.122644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:44.122838 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:44.122679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:44.122838 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:44.122656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:44.122838 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:44.122766 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-25spq" podUID="4f98337e-6fd2-4d84-9c89-6d03bf764a33" Oct 30 21:22:44.122838 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:44.122828 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7xnk" podUID="f3f8276e-9414-4129-8332-22be9c980723" Oct 30 21:22:44.123041 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:44.122881 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8r9s9" podUID="ab75e151-6ab3-4451-abf4-abb25a2e0463" Oct 30 21:22:45.962696 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:45.962449 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeReady" Oct 30 21:22:45.963143 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:45.962779 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:22:46.080028 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.079989 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bblgp"] Oct 30 21:22:46.147048 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.147009 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ckh6t"] Oct 30 21:22:46.147214 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.147150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.157534 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.157508 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bblgp"] Oct 30 21:22:46.157667 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.157632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:46.157667 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.157658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.157757 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.157638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:46.157884 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.157869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:46.191565 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.191543 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ckh6t"] Oct 30 21:22:46.200547 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.200522 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:22:46.201013 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.200995 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjpnc\"" Oct 30 21:22:46.201107 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.201027 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:22:46.210713 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.210690 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jj5qw\"" Oct 30 21:22:46.216058 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.215999 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:22:46.216058 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.216018 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:22:46.216169 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.216068 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:22:46.216169 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.216109 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:22:46.216169 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.216113 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:22:46.216169 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.216116 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:22:46.234414 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.234489 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kq9l\" (UniqueName: \"kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.234489 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.234549 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.234627 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.234675 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.234661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8ww\" (UniqueName: \"kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.240436 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.240413 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7rrvk\"" Oct 30 21:22:46.240436 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.240414 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:22:46.240544 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.240527 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr2gm\"" Oct 30 21:22:46.240544 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.240536 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wjtfq\"" Oct 30 21:22:46.335803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq9l\" (UniqueName: \"kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.335803 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.336053 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.336053 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.336053 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8ww\" (UniqueName: \"kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.336053 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.335952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.336053 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.336022 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:46.336263 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.336082 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:46.836063968 +0000 UTC m=+32.218823649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:46.336317 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.336273 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.336740 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.336720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.339954 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.339929 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.401217 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.401176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8ww\" (UniqueName: \"kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.401359 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.401266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq9l\" (UniqueName: \"kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.501059 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.500966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:46.595145 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.595083 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wsk2z"] Oct 30 21:22:46.603334 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.603308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.657895 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.657869 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wsk2z"] Oct 30 21:22:46.659708 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.659684 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:22:46.659846 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.659834 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nrzq\"" Oct 30 21:22:46.739373 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.739342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.739373 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.739376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msffp\" (UniqueName: \"kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.739579 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.739405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.739579 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.739504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.739579 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.739530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.741781 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.741750 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:22:46.741968 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.741946 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:22:46.742043 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.742030 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:22:46.767316 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.767250 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm"] Oct 30 21:22:46.773178 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.773157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:46.826201 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.826172 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:22:46.838331 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.838306 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm"] Oct 30 21:22:46.838516 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.838447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:46.839813 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.839778 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:46.839938 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.839826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.839938 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.839857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.839938 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.839874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.839938 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.839830 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.839963 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.839992 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.840024 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:47.340006062 +0000 UTC m=+32.722765744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : configmap "kube-rbac-proxy" not found Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.840048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:47.340031099 +0000 UTC m=+32.722790766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.840082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:46.840120 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.840104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.840336 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.840176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.840336 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.840200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msffp\" (UniqueName: \"kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.840336 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.840203 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:46.840336 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:46.840275 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:47.840263649 +0000 UTC m=+33.223023314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:46.840464 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.840411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.860791 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.860754 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5wvjr\"" Oct 30 21:22:46.896345 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.896313 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:22:46.904842 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.904813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msffp\" (UniqueName: \"kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:46.940396 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940363 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:22:46.940868 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:46.940868 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:46.940868 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:46.940868 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:46.940868 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.940737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllgh\" (UniqueName: \"kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh\") pod \"migrator-66df5d496b-trnxm\" (UID: \"3253c12b-7df9-4f9b-83d1-fdb9b55e860e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:46.941577 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.941558 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ckh6t"] Oct 30 21:22:46.944478 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:46.944443 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5686b209_4171_4cc7_8f73_b128ca2d068e.slice/crio-799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc WatchSource:0}: Error finding container 799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc: Status 404 returned error can't find the container with id 799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc Oct 30 21:22:46.953055 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.953032 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:22:46.990746 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:46.990721 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hzbf2\"" Oct 30 21:22:47.042136 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.042136 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.042136 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.042392 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.042392 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllgh\" (UniqueName: \"kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh\") pod \"migrator-66df5d496b-trnxm\" (UID: \"3253c12b-7df9-4f9b-83d1-fdb9b55e860e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:47.042549 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042527 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.042762 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.042745 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.043055 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.043036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.103207 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.103177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllgh\" (UniqueName: \"kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh\") pod \"migrator-66df5d496b-trnxm\" (UID: \"3253c12b-7df9-4f9b-83d1-fdb9b55e860e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:47.103344 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.103313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.162798 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.162760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:47.347376 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.347285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:47.347376 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.347337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:47.347583 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.347451 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:22:47.347583 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.347491 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:47.347583 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.347523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:48.34750523 +0000 UTC m=+33.730264912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : configmap "kube-rbac-proxy" not found Oct 30 21:22:47.347583 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.347562 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:48.347551931 +0000 UTC m=+33.730311611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:47.359181 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.359145 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ckh6t" event={"ID":"5686b209-4171-4cc7-8f73-b128ca2d068e","Type":"ContainerStarted","Data":"799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc"} Oct 30 21:22:47.390731 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.390699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:47.561817 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.561789 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:22:47.564852 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:47.564824 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec5d8d2_70c3_4a6a_a0ba_4f17f3c01300.slice/crio-a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f WatchSource:0}: Error finding container a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f: Status 404 returned error can't find the container with id a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f Oct 30 21:22:47.789340 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.789264 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm"] Oct 30 21:22:47.792029 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:47.792005 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3253c12b_7df9_4f9b_83d1_fdb9b55e860e.slice/crio-7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440 WatchSource:0}: Error finding container 7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440: Status 404 returned error can't find the container with id 7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440 Oct 30 21:22:47.851723 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:47.851684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:47.851896 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.851842 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:47.851940 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:47.851906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:49.851891411 +0000 UTC m=+35.234651076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:48.053845 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.053760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:48.054315 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.053861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:48.054315 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.054038 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:22:48.054315 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.054122 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs podName:4f98337e-6fd2-4d84-9c89-6d03bf764a33 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:20.054100595 +0000 UTC m=+65.436860283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs") pod "network-metrics-daemon-25spq" (UID: "4f98337e-6fd2-4d84-9c89-6d03bf764a33") : secret "metrics-daemon-secret" not found Oct 30 21:22:48.056709 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.056598 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:48.084077 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.084048 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fc7t"] Oct 30 21:22:48.095921 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.095893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.144673 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.144639 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fc7t"] Oct 30 21:22:48.149132 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.149111 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:22:48.149132 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.149114 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:22:48.149283 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.149240 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fz4rw\"" Oct 30 21:22:48.149315 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.149296 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:22:48.149350 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.149338 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:22:48.154759 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.154735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:48.157761 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.157740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:48.255845 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.255808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.256040 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.255870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.256040 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.255980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrt6j\" (UniqueName: \"kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.289122 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.289090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:48.306658 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.306634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:48.357054 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.357015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.357265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.357083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.357265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.357115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt6j\" (UniqueName: \"kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.357265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.357145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:48.357265 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.357180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:48.357484 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.357291 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:22:48.357484 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.357360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:50.357339777 +0000 UTC m=+35.740099461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : configmap "kube-rbac-proxy" not found Oct 30 21:22:48.358425 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.358299 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:48.358425 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:48.358392 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:50.358372385 +0000 UTC m=+35.741132074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:48.358830 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.358808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.380685 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.380598 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.383741 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.383639 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" event={"ID":"3253c12b-7df9-4f9b-83d1-fdb9b55e860e","Type":"ContainerStarted","Data":"7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440"} Oct 30 21:22:48.385952 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.385863 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" event={"ID":"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300","Type":"ContainerStarted","Data":"a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f"} Oct 30 21:22:48.420912 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.420883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt6j\" (UniqueName: \"kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.684546 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.684467 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8r9s9"] Oct 30 21:22:48.687951 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:48.687920 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab75e151_6ab3_4451_abf4_abb25a2e0463.slice/crio-9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b WatchSource:0}: Error finding container 9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b: Status 404 returned error can't find the container with id 9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b Oct 30 21:22:48.717391 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.717365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:48.740962 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:48.740927 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7xnk"] Oct 30 21:22:48.748943 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:48.747704 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f8276e_9414_4129_8332_22be9c980723.slice/crio-f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10 WatchSource:0}: Error finding container f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10: Status 404 returned error can't find the container with id f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10 Oct 30 21:22:49.119367 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:49.119325 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fc7t"] Oct 30 21:22:49.121884 ip-10-0-134-53 kubenswrapper[2579]: W1030 21:22:49.121847 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc563b6df_332a_4327_ae3b_4f633cf0f6af.slice/crio-e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491 WatchSource:0}: Error finding container e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491: Status 404 returned error can't find the container with id e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491 Oct 30 21:22:49.392908 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:49.392771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" event={"ID":"c563b6df-332a-4327-ae3b-4f633cf0f6af","Type":"ContainerStarted","Data":"e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491"} Oct 30 21:22:49.394530 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:49.394496 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q7xnk" event={"ID":"f3f8276e-9414-4129-8332-22be9c980723","Type":"ContainerStarted","Data":"f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10"} Oct 30 21:22:49.396791 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:49.396763 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r9s9" event={"ID":"ab75e151-6ab3-4451-abf4-abb25a2e0463","Type":"ContainerStarted","Data":"9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b"} Oct 30 21:22:49.869336 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:49.869244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:49.869504 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:49.869421 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:49.869570 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:49.869507 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:53.869484648 +0000 UTC m=+39.252244332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:50.375247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:50.374393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:50.375247 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:50.374517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:50.375247 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:50.374821 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:50.375247 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:50.374900 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:54.374884262 +0000 UTC m=+39.757643927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:50.376314 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:50.375708 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:22:50.376314 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:50.375771 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:54.37575345 +0000 UTC m=+39.758513115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : configmap "kube-rbac-proxy" not found Oct 30 21:22:50.405636 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:50.405218 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" event={"ID":"3253c12b-7df9-4f9b-83d1-fdb9b55e860e","Type":"ContainerStarted","Data":"7621eb5af3152e507e148d7b5cdab8d87e8aed939a630800a4051a12c706b303"} Oct 30 21:22:51.411197 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:51.411118 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" event={"ID":"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300","Type":"ContainerStarted","Data":"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914"} Oct 30 21:22:51.414087 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:51.413732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ckh6t" event={"ID":"5686b209-4171-4cc7-8f73-b128ca2d068e","Type":"ContainerStarted","Data":"a4e4c3e95c47f9d7a974a29825be4b6383d86631cf01878d8360845b779a74a8"} Oct 30 21:22:51.530974 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:51.530911 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" podStartSLOduration=3.404414396 podStartE2EDuration="5.530891615s" totalImagesPullingTime="2.126477219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.522759971 +0000 UTC m=+35.905519659" watchObservedRunningTime="2026-10-30 21:22:51.530891615 +0000 UTC m=+36.913651303" Oct 30 21:22:51.531924 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:51.531879 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" podStartSLOduration=2.013359309 podStartE2EDuration="5.531863028s" totalImagesPullingTime="3.518503719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:51.531477327 +0000 UTC m=+36.914237015" watchObservedRunningTime="2026-10-30 21:22:51.531863028 +0000 UTC m=+36.914622718" Oct 30 21:22:51.654895 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:51.654830 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ckh6t" podStartSLOduration=1.500915965 podStartE2EDuration="5.654810334s" totalImagesPullingTime="4.153894369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:51.653337959 +0000 UTC m=+37.036097650" watchObservedRunningTime="2026-10-30 21:22:51.654810334 +0000 UTC m=+37.037570020" Oct 30 21:22:53.905104 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:53.905060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:53.905550 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:53.905231 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:53.905550 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:53.905318 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:23:01.90529492 +0000 UTC m=+47.288054609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:54.408871 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.408822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:54.409053 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.408882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:54.409053 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:54.408987 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:54.409135 ip-10-0-134-53 kubenswrapper[2579]: E1030 21:22:54.409061 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:23:02.409041002 +0000 UTC m=+47.791800669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:54.409516 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.409495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:54.423967 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.423921 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" event={"ID":"c563b6df-332a-4327-ae3b-4f633cf0f6af","Type":"ContainerStarted","Data":"77faee2fef64c85f6e06a90ab4a812e3f2f5e423ab0e0ad5f8afd707e8ba6cde"} Oct 30 21:22:54.426020 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.425939 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8r9s9" event={"ID":"ab75e151-6ab3-4451-abf4-abb25a2e0463","Type":"ContainerStarted","Data":"6f8c4db47b6c9489c2419dd842dc5cfa922f177687f35a3ac61d1b7468b8d2f5"} Oct 30 21:22:54.426199 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.426173 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:54.544899 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.544738 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" podStartSLOduration=1.603451442 podStartE2EDuration="6.544724887s" totalImagesPullingTime="4.941273445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:54.544221888 +0000 UTC m=+39.926981577" watchObservedRunningTime="2026-10-30 21:22:54.544724887 +0000 UTC m=+39.927484574" Oct 30 21:22:54.662655 ip-10-0-134-53 kubenswrapper[2579]: I1030 21:22:54.662543 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8r9s9" podStartSLOduration=34.798446159 podStartE2EDuration="39.662527945s" totalImagesPullingTime="4.864081786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:54.661924848 +0000 UTC m=+40.044684536" watchObservedRunningTime="2026-10-30 21:22:54.662527945 +0000 UTC m=+40.045287631" Oct 30 21:22:55.256653 ip-10-0-134-53 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:22:55.269178 ip-10-0-134-53 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:22:55.269368 ip-10-0-134-53 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:22:55.269421 ip-10-0-134-53 systemd[1]: kubelet.service: Consumed 2.580s CPU time. Oct 30 21:22:55.292221 ip-10-0-134-53 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:22:55.390341 ip-10-0-134-53 kubenswrapper[4525]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:55.390341 ip-10-0-134-53 kubenswrapper[4525]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:22:55.390341 ip-10-0-134-53 kubenswrapper[4525]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:55.390341 ip-10-0-134-53 kubenswrapper[4525]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:22:55.392428 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392331 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.392428 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392429 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.392428 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392433 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392438 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392441 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392444 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392447 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392450 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392452 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392455 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392457 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392460 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392462 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392465 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392467 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392470 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392472 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392475 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392478 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392480 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392483 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.392546 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392493 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392495 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392498 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392500 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392503 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392505 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392508 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392511 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392513 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392516 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392518 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392521 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392523 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392526 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392529 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392532 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392535 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392538 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392541 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392544 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.393090 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392546 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392549 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392552 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392554 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392557 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392560 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392563 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392565 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392568 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392570 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392573 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392575 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392578 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392581 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392583 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392586 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392589 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392593 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392597 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.393592 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392600 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392617 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392620 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392624 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392626 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392629 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392632 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392634 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392637 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392640 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392643 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392645 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392648 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392652 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392654 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392658 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392661 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392664 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392666 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392669 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.394063 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392671 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392674 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392676 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392679 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392681 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392684 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392686 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392689 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392691 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392694 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392696 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392699 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392701 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392704 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392706 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392708 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392711 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392714 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392717 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.394536 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392721 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392726 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392729 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392733 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392736 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392738 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.392741 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393514 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393521 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393524 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393527 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393530 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393533 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393536 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393539 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393542 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393545 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393547 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393549 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393552 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.395046 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393555 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393558 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393560 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393563 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393566 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393568 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393571 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393573 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393576 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393579 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393581 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393584 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393586 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393589 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393592 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393594 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393597 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393599 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393621 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393624 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.395537 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393627 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393630 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393632 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393634 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393637 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393640 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393642 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393645 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393648 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393650 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393653 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393655 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393658 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393661 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393664 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393666 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393669 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393673 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393675 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393678 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.396050 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393682 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393685 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393688 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393691 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393693 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393696 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393698 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393701 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393703 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393706 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393708 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393711 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393714 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393718 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393724 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393726 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393729 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393732 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393734 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.396545 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393737 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393739 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393741 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393744 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393747 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393749 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393752 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393754 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393757 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393760 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393762 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393765 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393767 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393770 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393773 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393775 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393779 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393783 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393786 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.397028 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393788 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393791 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393793 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393796 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393798 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393801 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393803 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393806 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393810 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393812 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393815 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393819 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393821 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393824 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.393826 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394326 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394332 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394336 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394339 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.397477 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394342 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394345 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394347 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394350 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394352 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394355 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394358 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394360 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394363 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394365 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394368 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394370 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394372 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394375 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394377 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394380 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394382 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394384 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394387 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394389 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.397964 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394395 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394398 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394401 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394404 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394406 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394409 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394412 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394414 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394417 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394419 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394422 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394424 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394427 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394429 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394432 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394434 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394436 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394439 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394442 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.398471 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394444 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394446 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394449 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394451 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394454 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394456 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394458 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394461 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394463 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394466 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394469 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394471 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394474 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394476 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394479 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394482 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394484 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394487 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394489 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.398952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394492 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394495 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394497 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394500 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394502 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394505 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394507 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394510 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394512 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394514 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394517 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394519 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394522 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394524 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394526 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394529 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394531 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394534 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394536 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394539 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.399650 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394541 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394544 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394547 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394549 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394552 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394554 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394557 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394561 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394565 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394568 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394570 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394573 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394575 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394578 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394580 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394583 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394585 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394588 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394590 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.400476 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394592 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394595 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394597 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394600 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.394616 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394714 4525 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394721 4525 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394734 4525 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394738 4525 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394743 4525 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394746 4525 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394751 4525 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394755 4525 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394758 4525 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394761 4525 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394765 4525 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394768 4525 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394771 4525 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394774 4525 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394777 4525 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394782 4525 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394785 4525 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394794 4525 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394800 4525 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:22:55.401064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394803 4525 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394806 4525 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394809 4525 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394813 4525 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394817 4525 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394820 4525 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394823 4525 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394826 4525 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394829 4525 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394832 4525 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394835 4525 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394838 4525 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394841 4525 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394845 4525 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394848 4525 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394851 4525 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394853 4525 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394857 4525 flags.go:64] FLAG: --enable-server="true" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394859 4525 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394866 4525 flags.go:64] FLAG: --event-burst="100" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394869 4525 flags.go:64] FLAG: --event-qps="50" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394872 4525 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394875 4525 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394878 4525 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:22:55.401732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394882 4525 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394885 4525 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394888 4525 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394890 4525 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394895 4525 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394899 4525 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394902 4525 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394906 4525 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394909 4525 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394912 4525 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394915 4525 flags.go:64] FLAG: --feature-gates="" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394918 4525 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394921 4525 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394924 4525 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394927 4525 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394930 4525 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394933 4525 flags.go:64] FLAG: --help="false" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394936 4525 flags.go:64] FLAG: --hostname-override="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394939 4525 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394943 4525 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394946 4525 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394949 4525 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394952 4525 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:22:55.402537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394955 4525 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394959 4525 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394961 4525 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394964 4525 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394967 4525 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394970 4525 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394973 4525 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394976 4525 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394979 4525 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394982 4525 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394986 4525 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394989 4525 flags.go:64] FLAG: --lock-file="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394991 4525 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.394997 4525 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395001 4525 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395007 4525 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395010 4525 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395014 4525 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395017 4525 flags.go:64] FLAG: --logging-format="text" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395020 4525 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395023 4525 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395026 4525 flags.go:64] FLAG: --manifest-url="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395029 4525 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395033 4525 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395037 4525 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:22:55.403193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395041 4525 flags.go:64] FLAG: --max-pods="110" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395043 4525 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395046 4525 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395049 4525 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395052 4525 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395055 4525 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395058 4525 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395062 4525 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395068 4525 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395071 4525 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395074 4525 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395077 4525 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395080 4525 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395083 4525 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395086 4525 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395089 4525 flags.go:64] FLAG: --port="10250" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395092 4525 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395095 4525 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0472769f3faf3c528" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395098 4525 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395101 4525 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395105 4525 flags.go:64] FLAG: --register-node="true" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395111 4525 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395115 4525 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395117 4525 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395120 4525 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395123 4525 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:22:55.403899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395127 4525 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395130 4525 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395133 4525 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395136 4525 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395139 4525 flags.go:64] FLAG: --runonce="false" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395142 4525 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395145 4525 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395147 4525 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395150 4525 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395153 4525 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395156 4525 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395159 4525 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395162 4525 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395165 4525 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395168 4525 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395171 4525 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395173 4525 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395176 4525 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395179 4525 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395182 4525 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395187 4525 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395190 4525 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395193 4525 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395199 4525 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395202 4525 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:22:55.404681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395204 4525 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395210 4525 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395214 4525 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395217 4525 flags.go:64] FLAG: --v="2" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395221 4525 flags.go:64] FLAG: --version="false" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395226 4525 flags.go:64] FLAG: --vmodule="" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395230 4525 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395233 4525 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395403 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395407 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395410 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395413 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395415 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395420 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395424 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395426 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395429 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395432 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395434 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395437 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395440 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.405327 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395442 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395445 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395448 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395450 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395453 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395455 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395458 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395461 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395464 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395466 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395469 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395471 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395475 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395478 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395481 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395484 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395486 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395489 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395492 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.405907 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395495 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395497 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395500 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395502 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395505 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395507 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395510 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395512 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395515 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395517 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395520 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395522 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395526 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395530 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395532 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395535 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395537 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395540 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395543 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.406365 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395546 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395548 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395551 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395553 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395556 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395558 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395562 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395566 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395568 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395571 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395573 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395576 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395578 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395581 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395584 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395587 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395589 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395592 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395594 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395597 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.406825 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395599 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395615 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395618 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395621 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395623 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395626 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395628 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395631 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395634 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395636 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395639 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395641 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395644 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395646 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395649 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395651 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395654 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395656 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395660 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.407494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395663 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395666 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395668 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395671 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395674 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395676 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395679 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395681 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395684 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395687 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395690 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395692 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395695 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395697 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395699 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395702 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.408122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.395728 4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395769 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395774 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395778 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395783 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395786 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395789 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395791 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395794 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395796 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395799 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395802 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395804 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395806 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395809 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395813 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395816 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395819 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395822 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395824 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.408525 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395827 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395829 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395832 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395834 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395837 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395840 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395843 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395845 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395848 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395850 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395853 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395855 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395858 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395860 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395862 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395865 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395867 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395869 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395872 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.409074 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395875 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395878 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395880 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395883 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395885 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395887 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395890 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395892 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395896 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395898 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395902 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395904 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395907 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395909 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395912 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395914 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395917 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395920 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395922 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395925 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.409696 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395928 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395930 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395933 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395935 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395938 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395940 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395942 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395945 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395947 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395950 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395952 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395956 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395960 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395964 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395966 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395969 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395972 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395974 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395977 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.410264 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395979 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395984 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395986 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395990 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395993 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395995 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.395998 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396001 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396003 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396006 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396008 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396011 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396013 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396016 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396019 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396021 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396024 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396026 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396029 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.410936 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396031 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396034 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396036 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396039 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396041 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396044 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396046 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396049 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396052 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.396054 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.401765 4525 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.401782 4525 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.401803 4525 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.401808 4525 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401869 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401874 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401878 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401881 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401884 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.411389 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401886 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401889 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401892 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401894 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401897 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401899 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401902 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401905 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401908 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401911 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401913 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401916 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401919 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401921 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401924 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401926 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401929 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401931 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401934 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401936 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.411911 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401939 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401941 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401944 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401950 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401953 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401956 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401959 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401961 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401964 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401966 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401969 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401971 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401974 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401976 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401979 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401983 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401987 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401990 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401992 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401995 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.412388 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.401998 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402001 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402003 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402006 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402008 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402011 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402013 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402016 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402018 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402021 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402023 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402026 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402028 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402031 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402033 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402036 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402038 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402041 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402044 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402046 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.412931 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402049 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402052 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402055 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402057 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402060 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402062 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402065 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402068 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402070 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402072 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402075 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402077 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402080 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402084 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402088 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402091 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402093 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402096 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402099 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.413422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402101 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402104 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402107 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402109 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402112 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402114 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402117 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402119 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402122 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402124 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402127 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402129 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402132 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402134 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402137 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402139 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402142 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402144 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402146 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402149 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.413920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402151 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402154 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.402181 4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402325 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402330 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402333 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402335 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402338 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402341 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402344 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402346 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402349 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402351 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402354 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402356 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402359 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.414437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402361 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402364 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402366 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402369 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402372 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402375 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402377 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402380 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402382 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402385 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402388 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402390 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402393 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402395 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402398 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402400 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402402 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402405 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402407 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.414883 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402413 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402416 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402418 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402421 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402423 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402426 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402429 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402431 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402434 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402436 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402439 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402441 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402444 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402446 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402453 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402456 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402458 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402461 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402463 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.415437 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402466 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402468 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402471 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402473 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402476 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402478 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402481 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402485 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402489 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402493 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402496 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402499 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402502 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402506 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402509 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402511 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402514 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402516 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402519 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.416075 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402521 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402524 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402527 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402529 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402532 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402534 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402537 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402540 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402542 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402545 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402548 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402550 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402553 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402555 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402558 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402560 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402563 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402565 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402568 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402570 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.416548 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402573 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402575 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402578 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402580 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402582 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402585 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402589 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402591 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402593 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402596 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402600 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402616 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402619 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402622 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402625 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402628 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.417067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.402652 4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402684 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402688 4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402691 4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402694 4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402697 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402699 4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402702 4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402704 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402707 4525 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402709 4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402712 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402715 4525 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402717 4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402720 4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402722 4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402725 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402727 4525 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402730 4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402733 4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:22:55.417517 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402735 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402738 4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402742 4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402744 4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402747 4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402749 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402752 4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402756 4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402759 4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402761 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402764 4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402767 4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402770 4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402772 4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402775 4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402778 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402780 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402783 4525 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402785 4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:22:55.418307 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402788 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402790 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402793 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402795 4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402797 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402800 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402802 4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402804 4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402807 4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402809 4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402812 4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402814 4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402817 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402819 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402821 4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402825 4525 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402827 4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402830 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402832 4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:22:55.418999 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402836 4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402842 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402846 4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402849 4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402852 4525 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402855 4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402858 4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402861 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402864 4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402867 4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402869 4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402872 4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402874 4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402877 4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402880 4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402882 4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402885 4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402887 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402889 4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:22:55.419475 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402892 4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402894 4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402897 4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402899 4525 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402902 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402904 4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402907 4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402909 4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402912 4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402915 4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402918 4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402920 4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402923 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402926 4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402931 4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402934 4525 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402936 4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402939 4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402941 4525 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402943 4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:22:55.420077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402950 4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402953 4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402955 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402958 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402960 4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402963 4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402965 4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402967 4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402970 4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:55.402972 4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.403035 4525 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404052 4525 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404110 4525 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404300 4525 server.go:1014] "Starting client certificate rotation" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404449 4525 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404501 4525 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:17:15 +0000 UTC" deadline="2028-04-16 11:41:03.875825935 +0000 UTC" Oct 30 21:22:55.420581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404539 4525 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12806h18m8.471288408s" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.404793 4525 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.405855 4525 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.407358 4525 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.408945 4525 log.go:25] "Validated CRI v1 image API" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.409783 4525 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:22:55.421236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.411335 4525 fs.go:135] Filesystem UUIDs: map[5ece9ed1-1ef7-4ec3-9c2f-c330abd1f23c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7236199-0d43-45f8-a3d5-785967c5162c:/dev/nvme0n1p3] Oct 30 21:22:55.421520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.411356 4525 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f515ccd-ec8c-4407-8a96-72f3cf960b05/volumes/kubernetes.io~projected/kube-api-access-g57q6:{mountpoint:/var/lib/kubelet/pods/1f515ccd-ec8c-4407-8a96-72f3cf960b05/volumes/kubernetes.io~projected/kube-api-access-g57q6 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3253c12b-7df9-4f9b-83d1-fdb9b55e860e/volumes/kubernetes.io~projected/kube-api-access-cllgh:{mountpoint:/var/lib/kubelet/pods/3253c12b-7df9-4f9b-83d1-fdb9b55e860e/volumes/kubernetes.io~projected/kube-api-access-cllgh major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~projected/kube-api-access-bbpq5:{mountpoint:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~projected/kube-api-access-bbpq5 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~projected/kube-api-access-gbhvt:{mountpoint:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~projected/kube-api-access-gbhvt major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f98337e-6fd2-4d84-9c89-6d03bf764a33/volumes/kubernetes.io~projected/kube-api-access-9z2q8:{mountpoint:/var/lib/kubelet/pods/4f98337e-6fd2-4d84-9c89-6d03bf764a33/volumes/kubernetes.io~projected/kube-api-access-9z2q8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~projected/kube-api-access-6kq9l:{mountpoint:/var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~projected/kube-api-access-6kq9l major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68ab472e-ac86-4e2a-9939-148e357f191b/volumes/kubernetes.io~projected/kube-api-access-rj28h:{mountpoint:/var/lib/kubelet/pods/68ab472e-ac86-4e2a-9939-148e357f191b/volumes/kubernetes.io~projected/kube-api-access-rj28h major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7edb46bb-cfe0-4220-8843-c30652a67b40/volumes/kubernetes.io~projected/kube-api-access-2zzlp:{mountpoint:/var/lib/kubelet/pods/7edb46bb-cfe0-4220-8843-c30652a67b40/volumes/kubernetes.io~projected/kube-api-access-2zzlp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85e468f3-d3a6-4f2d-8964-f18c273b0db3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/85e468f3-d3a6-4f2d-8964-f18c273b0db3/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/975bc6cc-1f63-4ae5-ac26-771225c652b4/volumes/kubernetes.io~projected/kube-api-access-sqg6t:{mountpoint:/var/lib/kubelet/pods/975bc6cc-1f63-4ae5-ac26-771225c652b4/volumes/kubernetes.io~projected/kube-api-access-sqg6t major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98b136c6-693a-44bf-b141-b179fbbdc441/volumes/kubernetes.io~projected/kube-api-access-fx8ww:{mountpoint:/var/lib/kubelet/pods/98b136c6-693a-44bf-b141-b179fbbdc441/volumes/kubernetes.io~projected/kube-api-access-fx8ww major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a817f0e-4473-409d-9638-459e8b0a066f/volumes/kubernetes.io~projected/kube-api-access-8xvsv:{mountpoint:/var/lib/kubelet/pods/9a817f0e-4473-409d-9638-459e8b0a066f/volumes/kubernetes.io~projected/kube-api-access-8xvsv major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab75e151-6ab3-4451-abf4-abb25a2e0463/volumes/kubernetes.io~projected/kube-api-access-wxsxx:{mountpoint:/var/lib/kubelet/pods/ab75e151-6ab3-4451-abf4-abb25a2e0463/volumes/kubernetes.io~projected/kube-api-access-wxsxx major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~projected/kube-api-access-jrt6j:{mountpoint:/var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~projected/kube-api-access-jrt6j major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~secret/signing-key major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed19e99d-4341-4066-bb33-4fd4361b795a/volumes/kubernetes.io~projected/kube-api-access-h8gl4:{mountpoint:/var/lib/kubelet/pods/ed19e99d-4341-4066-bb33-4fd4361b795a/volumes/kubernetes.io~projected/kube-api-access-h8gl4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300/volumes/kubernetes.io~projected/kube-api-access-g87ch:{mountpoint:/var/lib/kubelet/pods/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300/volumes/kubernetes.io~projected/kube-api-access-g87ch major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3f8276e-9414-4129-8332-22be9c980723/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f3f8276e-9414-4129-8332-22be9c980723/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f/volumes/kubernetes.io~projected/kube-api-access-msffp:{mountpoint:/var/lib/kubelet/pods/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f/volumes/kubernetes.io~projected/kube-api-access-msffp major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c2cd54878c90e0fa68579c7bc8f6e0bbe4616ae9112fe08c91eab552a5575437/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d87e928623023847552c2945fae6270e8c7d6a252d4771f945f074430f4dc097/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a92bee622171bf5776fd24f6d17f87c479713e8911f5eb4835c48870b0c06967/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/293f19d48437f044b8dd7ca14a7b44087b377bd659b3cf4398026ae2e83ce351/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d1d7676a865b9abf56d6608af0807ad28a939c343361c0df708913c670508aea/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ce59117526e249271e6f05eac27150bba37b4887f94a44009786bc1ce6ed0460/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b3c5289c8e30150fe1ac1cc7493aeb8b6d74992e080271753be655cccd7171ec/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6e76756d1d87e0a5d2a3020a5b65df64ee1a9e31406d64f771d9d3d18876900e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c329d50049ccf535353916eb0039e039f821dbb249aa26f957cbfe7bc5c725c3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/0637ab312e666c24fa378b46f04880e5e7b394358c525622f6602b0cde128f30/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/2edb2f92f19f49e3792649773ff509617e9d4316b3fca8f95fd2a8190f492be0/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9ea3c4379d43d7c093782bd9d9673c9fe7cbe8e3d654a7279e0e4fb1f758ac51/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c0f865b2ee94941685fe6e0891536a6f19b4b66a35b4aa761f89c4705d33a52f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/4006a3592ae4d8cdb7dc3b4df7024712cebf65029c92e86ec42ae9ac7f4f95e4/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b9a509fadc215e498e27f5428a6f5677550d82efeec2c527fb82f94cff5cd7f4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/02b7e32eec07b08d36e285a4ab4360a71aa852c0285cc00c4e3fe67b06b45873/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9e4b2494818ae2ed7f971cf3daba8ae453bca911d84182870bdd27b22324a57b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3de79a3d570ddb4b16b3235d28d79ac60356fdc77d7aa5e81007ffc42ca4890a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/17cc0cd7b7e5f2e0a534445dd5da05e9abbc5c403e4d5a1f6e5dcf291b7d58c8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/41e8deb0e0ade4f2184f2534626a4f7de68b8de72f398cf094352e7c723d7f07/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-193:{mountpoint:/var/lib/containers/storage/overlay/78ca88fb9b6dc5d90521aec148ba96bd5f4228ebdc9692f6b09794003a565317/merged major:0 minor:193 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/37f03af32a6cd6b94ce53642d938ba2c2c807a3fc4a32ae489b9de7b6711754a/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2944469d600ec992e2f4cd91c54dac7c6693a2c5b0b13db67d6f35f386235da8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f86676e145cb8f3a7a1a93716a5deeecd33fc1ae5eefa3578cdfa5fc478b92ef/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b6c246033e383a0d18a240b0da1600b00d6c3f2a311890f55c3b4580b77480a6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/b2174ff4fb48c4881d6ca19b0f931ddb43cec22168563ab31b4765b25bbcdb45/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/087da273d938e4e130e03206786f6f19038b13168d8af9ae5ae10070bd2477de/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/9532caa15233e67d412ffdf13a1537c16e303ebba7ee9e744f3b483c2c6b3385/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/cc4bb67a68f9b78c3cbbfeffa0de264c9f853d109dc5088113aeea2aadf40d7c/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e3489b6ff05b533b7e1a31b48846de5ffd9ed986f042807dd6e9add62f03d6ef/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c5c32a9d53e7b03b5c69151478614455b339cf742a61cca4929702ecef21c895/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/25cc02cb477253e618729f2bb0b1f94018321221157cbf2830c3d51578b74f6e/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/db4ad055c8e2aa0d84b749d01c5058d7bda19c350a1a6b21c03e42f12265b458/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/6973db2784b016cec559453e448c756b0bcb2156f3f332cd8c29260d3f636b73/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a154431adff9e6bec996ffb808fd4d0bade069b84d865807cd77a5f83b46f456/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d1d020db3f9188e906a0caa8385651cb3e1a030691a8d173bd5e86a61662d3bd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/eec9944b1dcf6228b7ed8f8e37286cd94fb8f4eb049b969b06817939e360e3d3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9999ee30a34315687a3f25abf8145d0de98668ac39092f4610a058e42aac4bb0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/957ef9fab9f6e65bfa20559f1b82d356f494e800c04d8f56320648218b6bed17/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2e2421dcfd1c3a9a5892d89b019d512a47e161884b661b389e8a33efed65ec8e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/fd8155705c02dd746ec8f5289cc8e62d746bf823fdc4ea3a9f02181ca5dce0c2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/67cc883dc9311c2f2dd95898c9af6b3e451788089cdfc6865eb2e68ea1193d3b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8071722e04aab834554ba637bb249c38b0b8aba36eb7fb737e4701657c0f0181/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419012 4525 manager.go:217] Machine: {Timestamp:2026-10-30 21:22:55.418215324 +0000 UTC m=+0.092261853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107801 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9c9728a077a88b828b933d4cf503 SystemUUID:ec2b9c97-28a0-77a8-8b82-8b933d4cf503 BootID:c044d4ec-51d6-44bc-bbe2-11214d5ea0ed Filesystems:[{Device:/var/lib/kubelet/pods/4f98337e-6fd2-4d84-9c89-6d03bf764a33/volumes/kubernetes.io~projected/kube-api-access-9z2q8 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab75e151-6ab3-4451-abf4-abb25a2e0463/volumes/kubernetes.io~projected/kube-api-access-wxsxx DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23617941b7611fd35dbfe47bdd8694e3373d88012ffa2ad0651987443d2046cf/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/975bc6cc-1f63-4ae5-ac26-771225c652b4/volumes/kubernetes.io~projected/kube-api-access-sqg6t DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a817f0e-4473-409d-9638-459e8b0a066f/volumes/kubernetes.io~projected/kube-api-access-8xvsv DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7d445e08781c1c81c2cce6f185531e6ff3b53892b63c287ad756343b4519d10/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7edb46bb-cfe0-4220-8843-c30652a67b40/volumes/kubernetes.io~projected/kube-api-access-2zzlp DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98b136c6-693a-44bf-b141-b179fbbdc441/volumes/kubernetes.io~projected/kube-api-access-fx8ww DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~projected/kube-api-access-gbhvt DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-193 DeviceMajor:0 DeviceMinor:193 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f/volumes/kubernetes.io~projected/kube-api-access-msffp DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300/volumes/kubernetes.io~projected/kube-api-access-g87ch DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68ab472e-ac86-4e2a-9939-148e357f191b/volumes/kubernetes.io~projected/kube-api-access-rj28h DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ad4d4b6be0e7629e54b477d985a7986b838875d1bcf9e6a345f9a3c7c1f2dbb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d80bf61a72fbe67b4ca76581a7d75c95934c8959f642e8bb800072b1fb2eeaf7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf01ef351181dbf415c5ed9103ce3d6b2186560ad7e90e429037515306defa42/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5686b209-4171-4cc7-8f73-b128ca2d068e/volumes/kubernetes.io~projected/kube-api-access-6kq9l DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af80365cb14106a6bfb60a70a5dcab071ef27f583ed2108b2a94535fd203f4d9/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/799ce1d7b05bbedd54617d0d99f42bbde9f9cd81f8fdbb0414648e1fea24b3bc/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3253c12b-7df9-4f9b-83d1-fdb9b55e860e/volumes/kubernetes.io~projected/kube-api-access-cllgh DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5db0561547834ac0be5a0308988d105698bd7dbe4118c84f1bd00a7f98d1491/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~projected/kube-api-access-bbpq5 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed19e99d-4341-4066-bb33-4fd4361b795a/volumes/kubernetes.io~projected/kube-api-access-h8gl4 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3f8276e-9414-4129-8332-22be9c980723/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9baf39438cb2dae2c3fafcdaa2830d6539057d87725cbbce127600b62a220b6b/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a7c468c11cbbb8915d0b236f260bd8df3d259965aca0b941171d12728d58f57/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8deb46f4bc2e1af39b194a149a6be3d478142599b95eeee7522ad6e8ab7a870/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f515ccd-ec8c-4407-8a96-72f3cf960b05/volumes/kubernetes.io~projected/kube-api-access-g57q6 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dc365c44812df3c40a4ac07a8ab13b55d39f99c609bb75b11018f61bc1a8440/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/85e468f3-d3a6-4f2d-8964-f18c273b0db3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c563b6df-332a-4327-ae3b-4f633cf0f6af/volumes/kubernetes.io~projected/kube-api-access-jrt6j DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1f0e1eb259b0f4422a67af6cb5a3151b5ea9a2723e61b30eb95e9f737855121/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36343aa1-c487-43ea-9f03-9d639fad9893/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:799ce1d7b05bbed MacAddress:76:00:a5:b6:81:61 Speed:10000 Mtu:8901} {Name:7dc365c44812df3 MacAddress:ae:e0:ed:42:cb:f1 Speed:10000 Mtu:8901} {Name:9baf39438cb2dae MacAddress:fa:93:9c:10:d7:46 Speed:10000 Mtu:8901} {Name:a4bd4f0e8bcbcc1 MacAddress:d2:b8:cd:69:7e:fa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:68:7b:53:c0:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:9d:ea:df:26:47 Speed:0 Mtu:8901} {Name:e5db0561547834a MacAddress:ce:61:4b:e2:9e:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:68:7b:53:c0:f9 Speed:0 Mtu:9001} {Name:f7d445e08781c1c MacAddress:42:cd:41:72:7e:7b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8a:0c:33:3e:b4:28 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:21:23:03:ad:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419327 4525 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419389 4525 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419792 4525 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419824 4525 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-53.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419977 4525 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419984 4525 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.419995 4525 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420022 4525 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420122 4525 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420292 4525 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420410 4525 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420454 4525 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420465 4525 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420474 4525 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420483 4525 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.420492 4525 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.421241 4525 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:22:55.421810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.421253 4525 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:55.422799 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.421931 4525 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:22:55.425251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.424714 4525 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:22:55.425251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.424730 4525 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:22:55.425251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.424892 4525 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:22:55.425251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425159 4525 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:22:55.425251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425200 4525 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425338 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425362 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425374 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425406 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425418 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425430 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425441 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425456 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425471 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:22:55.425497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425483 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:22:55.425816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425510 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:22:55.425816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425528 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:22:55.425816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425569 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:22:55.425816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.425581 4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:22:55.429196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.429178 4525 server.go:1263] "Started kubelet" Oct 30 21:22:55.430357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.429268 4525 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:22:55.430357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.429931 4525 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:22:55.430357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.430003 4525 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:22:55.430357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.430300 4525 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:22:55.430588 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.430403 4525 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:22:55.430371 ip-10-0-134-53 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:22:55.433047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.432956 4525 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:22:55.434319 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434283 4525 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:22:55.434475 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.434438 4525 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:55.434673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434516 4525 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:22:55.434673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434533 4525 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:22:55.434832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434679 4525 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:22:55.434832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434498 4525 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:17:16 +0000 UTC" deadline="2028-06-23 07:39:03.632540979 +0000 UTC" Oct 30 21:22:55.434832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.434723 4525 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14434h16m8.19782377s" Oct 30 21:22:55.435591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.435559 4525 factory.go:55] Registering systemd factory Oct 30 21:22:55.435591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.435582 4525 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:22:55.437069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437048 4525 factory.go:153] Registering CRI-O factory Oct 30 21:22:55.437069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437071 4525 factory.go:223] Registration of the crio container factory successfully Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437065 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec" volumeName="kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437108 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f98337e-6fd2-4d84-9c89-6d03bf764a33" volumeName="kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437124 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5686b209-4171-4cc7-8f73-b128ca2d068e" volumeName="kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437135 4525 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437143 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975bc6cc-1f63-4ae5-ac26-771225c652b4" volumeName="kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437156 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab75e151-6ab3-4451-abf4-abb25a2e0463" volumeName="kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437164 4525 factory.go:103] Registering Raw factory Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437173 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed19e99d-4341-4066-bb33-4fd4361b795a" volumeName="kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437185 4525 manager.go:1199] Started watching for new ooms in manager Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437189 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f515ccd-ec8c-4407-8a96-72f3cf960b05" volumeName="kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:22:55.437204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437202 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36343aa1-c487-43ea-9f03-9d639fad9893" volumeName="kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437222 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85e468f3-d3a6-4f2d-8964-f18c273b0db3" volumeName="kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437234 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85e468f3-d3a6-4f2d-8964-f18c273b0db3" volumeName="kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437251 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b136c6-693a-44bf-b141-b179fbbdc441" volumeName="kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437276 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a817f0e-4473-409d-9638-459e8b0a066f" volumeName="kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437294 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c563b6df-332a-4327-ae3b-4f633cf0f6af" volumeName="kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437311 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed19e99d-4341-4066-bb33-4fd4361b795a" volumeName="kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437338 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f515ccd-ec8c-4407-8a96-72f3cf960b05" volumeName="kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437351 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec" volumeName="kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437368 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68ab472e-ac86-4e2a-9939-148e357f191b" volumeName="kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437381 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b136c6-693a-44bf-b141-b179fbbdc441" volumeName="kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437394 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c563b6df-332a-4327-ae3b-4f633cf0f6af" volumeName="kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437411 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" volumeName="kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437425 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3f8276e-9414-4129-8332-22be9c980723" volumeName="kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437441 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f42d54d0-0936-47a7-a7b2-a37a1ef64c1f" volumeName="kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437453 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f515ccd-ec8c-4407-8a96-72f3cf960b05" volumeName="kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437469 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36343aa1-c487-43ea-9f03-9d639fad9893" volumeName="kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437486 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68ab472e-ac86-4e2a-9939-148e357f191b" volumeName="kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437498 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" volumeName="kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437517 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f42d54d0-0936-47a7-a7b2-a37a1ef64c1f" volumeName="kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437529 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98b136c6-693a-44bf-b141-b179fbbdc441" volumeName="kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437552 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec" volumeName="kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437566 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7edb46bb-cfe0-4220-8843-c30652a67b40" volumeName="kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437572 4525 manager.go:319] Starting recovery of all containers Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437582 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed19e99d-4341-4066-bb33-4fd4361b795a" volumeName="kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config" seLinuxMountContext="" Oct 30 21:22:55.437749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.437593 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f42d54d0-0936-47a7-a7b2-a37a1ef64c1f" volumeName="kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438182 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3253c12b-7df9-4f9b-83d1-fdb9b55e860e" volumeName="kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438212 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36343aa1-c487-43ea-9f03-9d639fad9893" volumeName="kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438228 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c563b6df-332a-4327-ae3b-4f633cf0f6af" volumeName="kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438247 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" volumeName="kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438262 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" volumeName="kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438278 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f515ccd-ec8c-4407-8a96-72f3cf960b05" volumeName="kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438300 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5686b209-4171-4cc7-8f73-b128ca2d068e" volumeName="kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438316 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a817f0e-4473-409d-9638-459e8b0a066f" volumeName="kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438336 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975bc6cc-1f63-4ae5-ac26-771225c652b4" volumeName="kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438351 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36343aa1-c487-43ea-9f03-9d639fad9893" volumeName="kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438366 4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36343aa1-c487-43ea-9f03-9d639fad9893" volumeName="kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438384 4525 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:22:55.438653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.438394 4525 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:22:55.464158 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.464124 4525 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:22:55.465647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.465617 4525 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:22:55.465647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.465642 4525 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:22:55.465845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.465665 4525 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:22:55.465845 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.465707 4525 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:22:55.501130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.501095 4525 generic.go:356] "Generic (PLEG): container finished" podID="d7d17c7cfc239df4f9cffce0fad7645b" containerID="b972b111fe259520d0c42a525702f04d82147e347a5ba4dfbd24bf13c2dd0ae0" exitCode=0 Oct 30 21:22:55.510790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510722 4525 manager.go:324] Recovery completed Oct 30 21:22:55.510927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510794 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="eb78fc763a968f6a75ed2ccc7635ad4923056cd3c87db3efa3002482000f95e4" exitCode=0 Oct 30 21:22:55.510927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510816 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="facfdb94354216ec46545e54a78131f669e5762434dec67b85af3ea9c4c542dd" exitCode=0 Oct 30 21:22:55.510927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510825 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="984f1626f5bbda06d5d555f088c4aaffcfe6d578b44beb04c64c4557e947c7c8" exitCode=0 Oct 30 21:22:55.510927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510832 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="d9f5587bfd2590c17246a89eebacd544597b33b7425a9a7b0bd15c8b82d9562f" exitCode=0 Oct 30 21:22:55.510927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510839 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="555f32ef815b4f31558ee82a5c6fdbc386fb1d608104c1dd66686e282f4e9960" exitCode=0 Oct 30 21:22:55.511169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.510847 4525 generic.go:356] "Generic (PLEG): container finished" podID="1f515ccd-ec8c-4407-8a96-72f3cf960b05" containerID="9dda932121cbf9c2005b8535018741af122300d114cc23033f914fcfce8a74b4" exitCode=0 Oct 30 21:22:55.530903 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.530227 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.532043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.532008 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.532165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.532052 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.532165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.532068 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.533665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533650 4525 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:22:55.533665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533661 4525 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:22:55.533770 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533668 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.533770 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533677 4525 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:22:55.533834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533800 4525 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:22:55.533834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533811 4525 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:22:55.533889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533835 4525 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:22:55.533889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533843 4525 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:22:55.533889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533849 4525 policy_none.go:50] "Start" Oct 30 21:22:55.533889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533858 4525 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:22:55.533889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.533866 4525 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:22:55.534025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534013 4525 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:22:55.534053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534028 4525 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:22:55.534053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534037 4525 policy_none.go:44] "Start" Oct 30 21:22:55.534527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534492 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.534527 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.534519 4525 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:55.534636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534539 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.534636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.534552 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.538277 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.538247 4525 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:22:55.541437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.541421 4525 manager.go:341] "Starting Device Plugin manager" Oct 30 21:22:55.541551 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.541456 4525 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:22:55.541551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.541469 4525 server.go:87] "Starting device plugin registration server" Oct 30 21:22:55.541817 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.541802 4525 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:22:55.541880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.541818 4525 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:22:55.541956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.541937 4525 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:22:55.542124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.542071 4525 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:22:55.542233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.542140 4525 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:22:55.542233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.542147 4525 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:22:55.542338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.542223 4525 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:22:55.542119285Z","UUID":"66d2e75a-45da-4e94-913c-eb723daf6bf7","Handler":null,"Name":"","Endpoint":""} Oct 30 21:22:55.543384 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.543363 4525 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:22:55.543503 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.543402 4525 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" Oct 30 21:22:55.543847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.543830 4525 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:22:55.543951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.543860 4525 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:22:55.566487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.566000 4525 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal"] Oct 30 21:22:55.566661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.566584 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.567542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.567525 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.567680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.567561 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.567680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.567574 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.567911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.567897 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.568057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568043 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.568751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568734 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.568751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568749 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.568857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568768 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.568857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568785 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.568857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568768 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.568937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.568858 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.568968 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.568935 4525 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.569131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569102 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"ad6010805d267929c8af4d6a80598dea","Type":"ContainerStarted","Data":"f4146dd1f53b4bbea4b0fd7b7d6cd5d7f0b6df36cbf1b198189e2767948f23c0"} Oct 30 21:22:55.569167 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569145 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"ad6010805d267929c8af4d6a80598dea","Type":"ContainerStarted","Data":"680ecabd5f019e9a25764f2aa0fe3ce66562ba5d2d2cbfd0a8629692958de0f1"} Oct 30 21:22:55.569198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569164 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerStarted","Data":"dd1d0b352519250d02a0a092c165ed87fcc7456bf959056848ee32454a197963"} Oct 30 21:22:55.569198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569178 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerDied","Data":"b972b111fe259520d0c42a525702f04d82147e347a5ba4dfbd24bf13c2dd0ae0"} Oct 30 21:22:55.569198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569193 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" event={"ID":"d7d17c7cfc239df4f9cffce0fad7645b","Type":"ContainerStarted","Data":"b17368499711bb50a337213e569a3449a294d9e6e92bb1eb278f309d580a513e"} Oct 30 21:22:55.569297 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.569284 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.570486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.570460 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.570583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.570492 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.570583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.570508 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.570723 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.570706 4525 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-53.us-west-1.compute.internal\" not found" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.638650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.636912 4525 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:55.638650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.637248 4525 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:55.638650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.637527 4525 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:22:55.638650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.638289 4525 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:55.642227 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.642201 4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:22:55.644133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.643341 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:22:55.644133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.643372 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:22:55.644133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.643387 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:22:55.644133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.643433 4525 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.687653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.686884 4525 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:22:55.735207 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.735182 4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.738450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.738427 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.738563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.738453 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.738563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.738476 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.785232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.785151 4525 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.785232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.785226 4525 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.785841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.785819 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:22:55.785915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.785853 4525 setters.go:546] "Node became not ready" node="ip-10-0-134-53.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:22:55Z","lastTransitionTime":"2026-10-30T21:22:55Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:22:55.797653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.797629 4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:55.797761 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.797681 4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.797761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.797697 4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.804538 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.804513 4525 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-53.us-west-1.compute.internal" is owned by different node (oldNodeID="25e338ba-64cc-4221-8c13-e5d3ccb4d098", newNodeID=""), cleaning up... Oct 30 21:22:55.839343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839314 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.839343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839342 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.839526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839363 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.839526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839422 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.839526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839436 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad6010805d267929c8af4d6a80598dea-config\") pod \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"ad6010805d267929c8af4d6a80598dea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.839526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.839481 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7d17c7cfc239df4f9cffce0fad7645b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" (UID: \"d7d17c7cfc239df4f9cffce0fad7645b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:55.894236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:55.894206 4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:55.894400 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:55.894263 4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:56.421584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.421551 4525 apiserver.go:52] "Watching apiserver" Oct 30 21:22:56.513757 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.513731 4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:56.526512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.526481 4525 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:22:56.527008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.526987 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-2jlkf","openshift-image-registry/node-ca-pgkp4","openshift-multus/network-metrics-daemon-25spq","kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-wsk2z","kube-system/global-pull-secret-syncer-q7xnk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-8r9s9","openshift-network-operator/iptables-alerter-cx46g","openshift-cluster-node-tuning-operator/tuned-fsslh","openshift-ingress-canary/ingress-canary-ckh6t","openshift-insights/periodic-gathering-6bvq9-2cdd6","openshift-dns/node-resolver-czp4b","openshift-ovn-kubernetes/ovnkube-node-vh6nl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f","openshift-dns/dns-default-bblgp","openshift-multus/multus-6clh6","openshift-service-ca/service-ca-5fffc97bdf-4fc7t","openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm","kube-system/konnectivity-agent-pq7d9"] Oct 30 21:22:56.527594 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.527558 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.533644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.533624 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.534880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.534861 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.544844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544823 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.544942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544854 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.544942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544871 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.544942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544886 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.544942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544901 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.545088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.544972 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a817f0e-4473-409d-9638-459e8b0a066f-tmp-dir\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.545088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545003 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpq5\" (UniqueName: \"kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.545088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545032 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.545088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545077 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z2q8\" (UniqueName: \"kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.545238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545117 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.545238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545146 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.545238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545165 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhvt\" (UniqueName: \"kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545203 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.545379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545316 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.545379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545360 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.545379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545361 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b136c6-693a-44bf-b141-b179fbbdc441-tmp-dir\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.545472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545391 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.545472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545415 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.545472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545440 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545491 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545509 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.545660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545548 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.545660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545566 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.545660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545569 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-tmp\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545591 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.545660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545630 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545674 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545715 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545749 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545772 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545799 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrt6j\" (UniqueName: \"kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545811 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.545866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545821 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545889 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545916 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545941 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.545964 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546001 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546009 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-data-volume\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.546058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546030 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.546243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546070 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:56.546243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546101 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj28h\" (UniqueName: \"kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.546243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546129 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.546243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546223 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57q6\" (UniqueName: \"kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.546376 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546324 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.546376 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546342 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.546464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546379 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.546464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546404 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.546464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546451 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.546580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546493 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:56.546580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546518 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.546580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546556 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546586 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546629 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msffp\" (UniqueName: \"kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546648 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546665 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546691 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.546723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546713 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.546924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546745 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8ww\" (UniqueName: \"kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.546924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546857 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.546924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546891 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.546924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546909 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546923 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546941 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.547031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546976 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.547031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.546991 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:56.547031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547008 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.547162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547050 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.547162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547092 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547135 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547166 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.547244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547189 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.547244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547205 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kq9l\" (UniqueName: \"kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:56.547244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547220 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg6t\" (UniqueName: \"kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547246 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547271 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-tuned\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547277 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gl4\" (UniqueName: \"kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547303 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547322 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547338 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547352 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.547409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547406 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547435 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547464 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547484 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547500 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvsv\" (UniqueName: \"kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547534 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547560 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547586 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzlp\" (UniqueName: \"kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547615 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547632 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547662 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.547699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547688 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547718 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547785 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547821 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547846 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547883 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547908 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547945 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.547994 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllgh\" (UniqueName: \"kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh\") pod \"migrator-66df5d496b-trnxm\" (UID: \"3253c12b-7df9-4f9b-83d1-fdb9b55e860e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.548016 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.548047 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.548072 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.548135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.548096 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.548495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.548141 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.579325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.579298 4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:22:56.579476 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:56.579363 4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-53.us-west-1.compute.internal" Oct 30 21:22:56.588172 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588137 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:22:56.588367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588179 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:22:56.588367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588206 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nr2gm\"" Oct 30 21:22:56.588367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588218 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.588367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588355 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.588652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588370 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:22:56.588652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588393 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.588652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588626 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ntkmx\"" Oct 30 21:22:56.588810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588792 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:22:56.588810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588805 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gkfxf\"" Oct 30 21:22:56.589002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.588984 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jj5qw\"" Oct 30 21:22:56.591972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.591959 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.596329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.596313 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3f8276e-9414-4129-8332-22be9c980723-original-pull-secret\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.597405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.597391 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85e468f3-d3a6-4f2d-8964-f18c273b0db3-konnectivity-ca\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:56.597442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.597402 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85e468f3-d3a6-4f2d-8964-f18c273b0db3-agent-certs\") pod \"konnectivity-agent-pq7d9\" (UID: \"85e468f3-d3a6-4f2d-8964-f18c273b0db3\") " pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:22:56.635730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.635703 4525 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:22:56.638559 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.638536 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z2q8\" (UniqueName: \"kubernetes.io/projected/4f98337e-6fd2-4d84-9c89-6d03bf764a33-kube-api-access-9z2q8\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.648913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.648892 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.649012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.648925 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.649012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.648952 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.649012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.648977 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.649012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649000 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649016 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-dbus\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649024 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-socket-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649028 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649078 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975bc6cc-1f63-4ae5-ac26-771225c652b4-host\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649082 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649113 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-conf-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649117 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649131 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-bin\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649157 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-netns\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649160 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649186 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-kubernetes\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649194 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649207 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-systemd-units\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649218 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649246 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649255 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-systemd\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649273 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649297 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649311 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-registration-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649324 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649348 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649358 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-systemd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649337 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-socket-dir-parent\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649392 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-hostroot\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649398 4525 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649417 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649450 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649402 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-netns\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649472 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649476 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649478 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-system-cni-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649514 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649523 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysconfig\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649538 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-bin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649540 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649564 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-cni-netd\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649575 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649591 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649631 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649649 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.649647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649653 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-cni-multus\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649665 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-run-ovn\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649665 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649680 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-node-log\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649701 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649706 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-log-socket\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649717 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649743 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649772 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649798 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649802 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649816 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-host\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649825 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649852 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649868 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-var-lib-kubelet\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649915 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649993 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-sys\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.649993 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-conf\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650035 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650064 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650092 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-etc-kubernetes\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650124 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-device-dir\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650239 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.650486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650113 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-slash\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650506 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650522 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650538 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650555 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650570 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650592 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650623 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-run-ovn-kubernetes\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650636 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650640 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-var-lib-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650668 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cnibin\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650672 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650643 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-crio-socket\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650688 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-cnibin\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650691 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-etc-selinux\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650702 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650723 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-k8s-cni-cncf-io\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650728 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650733 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-lib-modules\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650734 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-host-run-multus-certs\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650762 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-host-kubelet\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650762 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650784 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650792 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36343aa1-c487-43ea-9f03-9d639fad9893-etc-openvswitch\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650800 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650814 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650829 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650869 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650874 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-run\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650893 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-modprobe-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650878 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650896 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650915 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a817f0e-4473-409d-9638-459e8b0a066f-hosts-file\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650922 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68ab472e-ac86-4e2a-9939-148e357f191b-host-slash\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650933 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-etc-sysctl-d\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650942 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.650982 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651009 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651034 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651039 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3f8276e-9414-4129-8332-22be9c980723-kubelet-config\") pod \"global-pull-secret-syncer-q7xnk\" (UID: \"f3f8276e-9414-4129-8332-22be9c980723\") " pod="kube-system/global-pull-secret-syncer-q7xnk" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651059 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651084 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7edb46bb-cfe0-4220-8843-c30652a67b40-sys-fs\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651098 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-var-lib-kubelet\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651131 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed19e99d-4341-4066-bb33-4fd4361b795a-system-cni-dir\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.651465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651374 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8ww\" (UniqueName: \"kubernetes.io/projected/98b136c6-693a-44bf-b141-b179fbbdc441-kube-api-access-fx8ww\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.652621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651825 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gl4\" (UniqueName: \"kubernetes.io/projected/ed19e99d-4341-4066-bb33-4fd4361b795a-kube-api-access-h8gl4\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.652621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651830 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvsv\" (UniqueName: \"kubernetes.io/projected/9a817f0e-4473-409d-9638-459e8b0a066f-kube-api-access-8xvsv\") pod \"node-resolver-czp4b\" (UID: \"9a817f0e-4473-409d-9638-459e8b0a066f\") " pod="openshift-dns/node-resolver-czp4b" Oct 30 21:22:56.652621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.651971 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f98337e-6fd2-4d84-9c89-6d03bf764a33-metrics-certs\") pod \"network-metrics-daemon-25spq\" (UID: \"4f98337e-6fd2-4d84-9c89-6d03bf764a33\") " pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.652621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.652139 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57q6\" (UniqueName: \"kubernetes.io/projected/1f515ccd-ec8c-4407-8a96-72f3cf960b05-kube-api-access-g57q6\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.674263 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674199 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.674263 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674252 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.674509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674268 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.674509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674281 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.674509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674491 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.674708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674696 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.674995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.674970 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.675125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675099 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675189 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675220 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675261 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675381 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675442 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675517 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:22:56.675633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.675626 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:22:56.678383 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.678357 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-env-overrides\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.678473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.678413 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.678675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.678656 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpq5\" (UniqueName: \"kubernetes.io/projected/36343aa1-c487-43ea-9f03-9d639fad9893-kube-api-access-bbpq5\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.678890 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.678871 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975bc6cc-1f63-4ae5-ac26-771225c652b4-serviceca\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.678944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.678922 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsxx\" (UniqueName: \"kubernetes.io/projected/ab75e151-6ab3-4451-abf4-abb25a2e0463-kube-api-access-wxsxx\") pod \"network-check-target-8r9s9\" (UID: \"ab75e151-6ab3-4451-abf4-abb25a2e0463\") " pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:22:56.680517 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.680499 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzlp\" (UniqueName: \"kubernetes.io/projected/7edb46bb-cfe0-4220-8843-c30652a67b40-kube-api-access-2zzlp\") pod \"aws-ebs-csi-driver-node-8bl4f\" (UID: \"7edb46bb-cfe0-4220-8843-c30652a67b40\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8bl4f" Oct 30 21:22:56.682692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.682678 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj28h\" (UniqueName: \"kubernetes.io/projected/68ab472e-ac86-4e2a-9939-148e357f191b-kube-api-access-rj28h\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.703726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.703701 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:22:56.707069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.707051 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-multus-daemon-config\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.723833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723804 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:22:56.723833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723818 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723841 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723863 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723871 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723890 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723900 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723919 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:22:56.723995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.723951 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724008 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724008 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724025 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724040 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724050 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724057 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724090 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724105 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:56.724224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724118 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:22:56.724565 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724412 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:22:56.724628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724599 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqzlq\"" Oct 30 21:22:56.724784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724770 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mbjr2\"" Oct 30 21:22:56.724831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724784 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:22:56.724831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724785 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:22:56.724831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.724785 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf4nn\"" Oct 30 21:22:56.725100 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725076 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7rrvk\"" Oct 30 21:22:56.725100 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725089 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fz4rw\"" Oct 30 21:22:56.725100 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725098 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vm9q9\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725120 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725127 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hzbf2\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725130 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5wvjr\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725162 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725181 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wjtfq\"" Oct 30 21:22:56.725256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725186 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gmgqb\"" Oct 30 21:22:56.725414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725361 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nrzq\"" Oct 30 21:22:56.725414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725389 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bcb\"" Oct 30 21:22:56.725414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725413 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjpnc\"" Oct 30 21:22:56.725496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725455 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lclr\"" Oct 30 21:22:56.725496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725472 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:22:56.725855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725836 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed19e99d-4341-4066-bb33-4fd4361b795a-cni-binary-copy\") pod \"multus-6clh6\" (UID: \"ed19e99d-4341-4066-bb33-4fd4361b795a\") " pod="openshift-multus/multus-6clh6" Oct 30 21:22:56.725924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.725888 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.726308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.726289 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.726435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.726423 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.726561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.726536 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98b136c6-693a-44bf-b141-b179fbbdc441-config-volume\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:56.726688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.726675 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.726728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.726679 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f515ccd-ec8c-4407-8a96-72f3cf960b05-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jlkf\" (UID: \"1f515ccd-ec8c-4407-8a96-72f3cf960b05\") " pod="openshift-multus/multus-additional-cni-plugins-2jlkf" Oct 30 21:22:56.727583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.727569 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5686b209-4171-4cc7-8f73-b128ca2d068e-cert\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:56.727721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.727707 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68ab472e-ac86-4e2a-9939-148e357f191b-iptables-alerter-script\") pod \"iptables-alerter-cx46g\" (UID: \"68ab472e-ac86-4e2a-9939-148e357f191b\") " pod="openshift-network-operator/iptables-alerter-cx46g" Oct 30 21:22:56.727800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.727780 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-config\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.728013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728000 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrt6j\" (UniqueName: \"kubernetes.io/projected/c563b6df-332a-4327-ae3b-4f633cf0f6af-kube-api-access-jrt6j\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.728121 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728109 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36343aa1-c487-43ea-9f03-9d639fad9893-ovn-node-metrics-cert\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.728381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728257 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.728381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728287 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c563b6df-332a-4327-ae3b-4f633cf0f6af-signing-key\") pod \"service-ca-5fffc97bdf-4fc7t\" (UID: \"c563b6df-332a-4327-ae3b-4f633cf0f6af\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fc7t" Oct 30 21:22:56.728511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728466 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36343aa1-c487-43ea-9f03-9d639fad9893-ovnkube-script-lib\") pod \"ovnkube-node-vh6nl\" (UID: \"36343aa1-c487-43ea-9f03-9d639fad9893\") " pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:56.728629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.728593 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhvt\" (UniqueName: \"kubernetes.io/projected/4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec-kube-api-access-gbhvt\") pod \"tuned-fsslh\" (UID: \"4abb6ccb-d1fa-45e4-88d7-84ed3a7a0eec\") " pod="openshift-cluster-node-tuning-operator/tuned-fsslh" Oct 30 21:22:56.729718 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:56.729705 4525 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:22:56.729781 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:56.729772 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls podName:98b136c6-693a-44bf-b141-b179fbbdc441 nodeName:}" failed. No retries permitted until 2026-10-30 21:22:57.229749195 +0000 UTC m=+1.903795736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls") pod "dns-default-bblgp" (UID: "98b136c6-693a-44bf-b141-b179fbbdc441") : secret "dns-default-metrics-tls" not found Oct 30 21:22:56.730154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.730140 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"periodic-gathering-6bvq9-2cdd6\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:22:56.732096 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:56.732079 4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:56.732173 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:56.732160 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:57.23213515 +0000 UTC m=+1.906181666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:56.732302 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.732285 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msffp\" (UniqueName: \"kubernetes.io/projected/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-kube-api-access-msffp\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:56.732394 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.732379 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg6t\" (UniqueName: \"kubernetes.io/projected/975bc6cc-1f63-4ae5-ac26-771225c652b4-kube-api-access-sqg6t\") pod \"node-ca-pgkp4\" (UID: \"975bc6cc-1f63-4ae5-ac26-771225c652b4\") " pod="openshift-image-registry/node-ca-pgkp4" Oct 30 21:22:56.732518 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.732500 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllgh\" (UniqueName: \"kubernetes.io/projected/3253c12b-7df9-4f9b-83d1-fdb9b55e860e-kube-api-access-cllgh\") pod \"migrator-66df5d496b-trnxm\" (UID: \"3253c12b-7df9-4f9b-83d1-fdb9b55e860e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-trnxm" Oct 30 21:22:56.734743 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.734729 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kq9l\" (UniqueName: \"kubernetes.io/projected/5686b209-4171-4cc7-8f73-b128ca2d068e-kube-api-access-6kq9l\") pod \"ingress-canary-ckh6t\" (UID: \"5686b209-4171-4cc7-8f73-b128ca2d068e\") " pod="openshift-ingress-canary/ingress-canary-ckh6t" Oct 30 21:22:56.828881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.828850 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-25spq" Oct 30 21:22:56.935264 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:56.935234 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ckh6t_5686b209-4171-4cc7-8f73-b128ca2d068e/serve-healthcheck-canary/0.log" Oct 30 21:22:57.246732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.246594 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-25spq"] Oct 30 21:22:57.251189 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:57.251147 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f98337e_6fd2_4d84_9c89_6d03bf764a33.slice/crio-2f5801bb2d0b80229687b640f16e3ac25cc029ddfae0278cc2ca69137557f776 WatchSource:0}: Error finding container 2f5801bb2d0b80229687b640f16e3ac25cc029ddfae0278cc2ca69137557f776: Status 404 returned error can't find the container with id 2f5801bb2d0b80229687b640f16e3ac25cc029ddfae0278cc2ca69137557f776 Oct 30 21:22:57.253368 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.253350 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:22:57.255473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.255449 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:57.255569 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.255493 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:57.255655 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:57.255590 4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:57.255692 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:57.255675 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:22:58.255657519 +0000 UTC m=+2.929704034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:57.257833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.257815 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98b136c6-693a-44bf-b141-b179fbbdc441-metrics-tls\") pod \"dns-default-bblgp\" (UID: \"98b136c6-693a-44bf-b141-b179fbbdc441\") " pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:57.434206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.434163 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bblgp" Oct 30 21:22:57.518691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.518659 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25spq" event={"ID":"4f98337e-6fd2-4d84-9c89-6d03bf764a33","Type":"ContainerStarted","Data":"2f5801bb2d0b80229687b640f16e3ac25cc029ddfae0278cc2ca69137557f776"} Oct 30 21:22:57.519563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.519541 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-trnxm_3253c12b-7df9-4f9b-83d1-fdb9b55e860e/migrator/0.log" Oct 30 21:22:57.564150 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.564126 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:22:57.833619 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:57.833524 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bblgp"] Oct 30 21:22:57.837267 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:22:57.837238 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b136c6_693a_44bf_b141_b179fbbdc441.slice/crio-0c80521b54037f4785e693a8817a88c0848d13e00eaaaf68dd1c204f28a2f14e WatchSource:0}: Error finding container 0c80521b54037f4785e693a8817a88c0848d13e00eaaaf68dd1c204f28a2f14e: Status 404 returned error can't find the container with id 0c80521b54037f4785e693a8817a88c0848d13e00eaaaf68dd1c204f28a2f14e Oct 30 21:22:58.264248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:58.264212 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:22:58.264432 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:58.264353 4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:22:58.264495 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:22:58.264448 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:23:00.264428468 +0000 UTC m=+4.938474985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:22:58.522308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:58.522227 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bblgp" event={"ID":"98b136c6-693a-44bf-b141-b179fbbdc441","Type":"ContainerStarted","Data":"0c80521b54037f4785e693a8817a88c0848d13e00eaaaf68dd1c204f28a2f14e"} Oct 30 21:22:59.533629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:59.533562 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25spq" event={"ID":"4f98337e-6fd2-4d84-9c89-6d03bf764a33","Type":"ContainerStarted","Data":"70b5591e44dbf706b2bd97f502fbf06cef43899a4a570857eff4334b30bf5691"} Oct 30 21:22:59.534073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:22:59.533684 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-25spq" event={"ID":"4f98337e-6fd2-4d84-9c89-6d03bf764a33","Type":"ContainerStarted","Data":"f7e15407dac8fb85b56eb18c2cc46ff29919e475aed7f44cb6910600ee9e9afb"} Oct 30 21:23:00.281670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:00.281569 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:23:00.281821 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:23:00.281716 4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:23:00.281821 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:23:00.281782 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls podName:f42d54d0-0936-47a7-a7b2-a37a1ef64c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:23:04.281765812 +0000 UTC m=+8.955812327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wsk2z" (UID: "f42d54d0-0936-47a7-a7b2-a37a1ef64c1f") : secret "insights-runtime-extractor-tls" not found Oct 30 21:23:00.538767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:00.538678 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bblgp" event={"ID":"98b136c6-693a-44bf-b141-b179fbbdc441","Type":"ContainerStarted","Data":"12ca76f42850128bb4a657f7321c4c11a4c54fc2897fdddafa43e336976d747d"} Oct 30 21:23:00.538767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:00.538736 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bblgp" event={"ID":"98b136c6-693a-44bf-b141-b179fbbdc441","Type":"ContainerStarted","Data":"142aefb1dd3fc11c3eb6f95f1cb79d1ec6e75c4a03801c4ad52259e17580ef4c"} Oct 30 21:23:00.993843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:00.993807 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:23:00.994128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:00.994113 4525 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:23:01.544217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:01.544179 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bblgp" Oct 30 21:23:01.739042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:01.739013 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vh6nl" Oct 30 21:23:04.310860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:04.310824 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:23:04.313239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:04.313214 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f42d54d0-0936-47a7-a7b2-a37a1ef64c1f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wsk2z\" (UID: \"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f\") " pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:23:04.335203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:04.335176 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wsk2z" Oct 30 21:23:04.703555 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:04.703525 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wsk2z"] Oct 30 21:23:04.706681 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:04.706649 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42d54d0_0936_47a7_a7b2_a37a1ef64c1f.slice/crio-b0b3cfc003267503007c7723a9af37133424861a2026ea8d2e9fefeb47ec6143 WatchSource:0}: Error finding container b0b3cfc003267503007c7723a9af37133424861a2026ea8d2e9fefeb47ec6143: Status 404 returned error can't find the container with id b0b3cfc003267503007c7723a9af37133424861a2026ea8d2e9fefeb47ec6143 Oct 30 21:23:05.559123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:05.559081 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsk2z" event={"ID":"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f","Type":"ContainerStarted","Data":"0e04c0f686111b4a1134493123074ee86350a7a2b59943d50bfdae60c11c1754"} Oct 30 21:23:05.559491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:05.559159 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsk2z" event={"ID":"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f","Type":"ContainerStarted","Data":"b0b3cfc003267503007c7723a9af37133424861a2026ea8d2e9fefeb47ec6143"} Oct 30 21:23:06.148324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:06.148297 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:23:06.181548 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:06.181518 4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-53.us-west-1.compute.internal" event="NodeReady" Oct 30 21:23:06.563543 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:06.563466 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsk2z" event={"ID":"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f","Type":"ContainerStarted","Data":"5a909a74a43b096130f82f7f7f46dc53fe2c2d5704f3f412148d88a26ca55f35"} Oct 30 21:23:07.421468 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.421437 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fddbbf884-7gkgc"] Oct 30 21:23:07.423469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.423451 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.527591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.527546 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:23:07.527591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.527588 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pdhmp\"" Oct 30 21:23:07.527917 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.527640 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:23:07.528464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.528445 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:23:07.535713 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535688 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-certificates\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.535829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535726 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-trusted-ca\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.535829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535759 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-bound-sa-token\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.535829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535806 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sqw\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-kube-api-access-r2sqw\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.535829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535828 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-tls\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.536013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535848 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-image-registry-private-configuration\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.536013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535865 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-ca-trust-extracted\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.536013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.535879 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-installation-pull-secrets\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.621189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.621152 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:23:07.628557 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.628522 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fddbbf884-7gkgc"] Oct 30 21:23:07.636134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636110 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sqw\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-kube-api-access-r2sqw\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636145 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-tls\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636167 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-image-registry-private-configuration\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636191 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-ca-trust-extracted\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636317 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-installation-pull-secrets\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636358 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-certificates\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636383 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-trusted-ca\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636417 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-bound-sa-token\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.636665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.636642 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-ca-trust-extracted\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.637292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.637268 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-certificates\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.637550 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.637526 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-trusted-ca\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.639161 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.639142 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-image-registry-private-configuration\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.639424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.639404 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-installation-pull-secrets\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.639511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.639496 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-registry-tls\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.698203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.698120 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-bound-sa-token\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.698348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.698303 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sqw\" (UniqueName: \"kubernetes.io/projected/0c75964e-1c00-43ce-ad8c-e3e5010f5d29-kube-api-access-r2sqw\") pod \"image-registry-6fddbbf884-7gkgc\" (UID: \"0c75964e-1c00-43ce-ad8c-e3e5010f5d29\") " pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:07.735735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:07.735702 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:08.162823 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.162791 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fddbbf884-7gkgc"] Oct 30 21:23:08.166077 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:08.166046 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c75964e_1c00_43ce_ad8c_e3e5010f5d29.slice/crio-9f0ebe42fc1e3fac84976d41a15223982c36706afd90dd176fa008e9e7dffeb2 WatchSource:0}: Error finding container 9f0ebe42fc1e3fac84976d41a15223982c36706afd90dd176fa008e9e7dffeb2: Status 404 returned error can't find the container with id 9f0ebe42fc1e3fac84976d41a15223982c36706afd90dd176fa008e9e7dffeb2 Oct 30 21:23:08.573528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.573431 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wsk2z" event={"ID":"f42d54d0-0936-47a7-a7b2-a37a1ef64c1f","Type":"ContainerStarted","Data":"96dee44872817f4ab6b9be2aea21a8c7531c0c48f760aebcc567c17e8b6638f8"} Oct 30 21:23:08.574949 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.574924 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" event={"ID":"0c75964e-1c00-43ce-ad8c-e3e5010f5d29","Type":"ContainerStarted","Data":"9f3786a891cf22c5994db26f9282115ae5f8e1e89661c6c7bfbededf1078f07b"} Oct 30 21:23:08.575085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.574968 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" event={"ID":"0c75964e-1c00-43ce-ad8c-e3e5010f5d29","Type":"ContainerStarted","Data":"9f0ebe42fc1e3fac84976d41a15223982c36706afd90dd176fa008e9e7dffeb2"} Oct 30 21:23:08.575085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.575000 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:08.812136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:08.812064 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" podStartSLOduration=1.812009399 podStartE2EDuration="1.812009399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:08.811546657 +0000 UTC m=+13.485593194" watchObservedRunningTime="2026-10-30 21:23:08.812009399 +0000 UTC m=+13.486055936" Oct 30 21:23:11.551754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:11.551727 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bblgp" Oct 30 21:23:19.273580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:19.273549 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pq7d9" Oct 30 21:23:22.841225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.841193 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-w9wjt-4qhsz"] Oct 30 21:23:22.844195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.844177 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:22.902563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.902519 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w9wjt-4qhsz"] Oct 30 21:23:22.930411 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.930353 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qsd\" (UniqueName: \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:22.930411 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.930414 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:22.930628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.930437 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:22.930628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:22.930553 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.031448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.031355 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qsd\" (UniqueName: \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.031448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.031391 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.031448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.031412 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.031448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.031443 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.031814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.031798 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.032100 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.032082 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.032361 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.032344 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.092688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.092656 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qsd\" (UniqueName: \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\") pod \"periodic-gathering-w9wjt-4qhsz\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.154511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.154479 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:23:23.521368 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.521322 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-w9wjt-4qhsz"] Oct 30 21:23:23.523947 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:23.523922 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b523b1_c52d_416d_8ddc_ac5d51b0b5b1.slice/crio-64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc WatchSource:0}: Error finding container 64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc: Status 404 returned error can't find the container with id 64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc Oct 30 21:23:23.621531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.621494 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" event={"ID":"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1","Type":"ContainerStarted","Data":"25466fae2ef76d4cd63be3955d8c45837b6212cd3770733e5a781206f4c0637b"} Oct 30 21:23:23.621745 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.621551 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" event={"ID":"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1","Type":"ContainerStarted","Data":"64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc"} Oct 30 21:23:23.742798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:23.742740 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" podStartSLOduration=1.742727229 podStartE2EDuration="1.742727229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:23.740088129 +0000 UTC m=+28.414134665" watchObservedRunningTime="2026-10-30 21:23:23.742727229 +0000 UTC m=+28.416773743" Oct 30 21:23:26.151291 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:26.151264 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8r9s9" Oct 30 21:23:26.977602 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:26.977568 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5gnht"] Oct 30 21:23:26.979800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:26.979782 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061654 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-tls\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061690 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-sys\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061714 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-textfile\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061812 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061852 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-wtmp\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.061926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061877 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.062069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061970 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-metrics-client-ca\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.062069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.061997 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt925\" (UniqueName: \"kubernetes.io/projected/bc1fc34a-ac92-4787-9859-2535ff564f73-kube-api-access-wt925\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.062069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.062056 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-root\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.074588 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.074561 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:23:27.086152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.086131 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:23:27.086303 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.086179 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pvd8m\"" Oct 30 21:23:27.124824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.124795 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:23:27.125006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.124989 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:23:27.138826 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.136457 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:23:27.138826 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.137012 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:23:27.162448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162412 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-textfile\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162452 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162471 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-wtmp\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162519 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162539 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-metrics-client-ca\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162554 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt925\" (UniqueName: \"kubernetes.io/projected/bc1fc34a-ac92-4787-9859-2535ff564f73-kube-api-access-wt925\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162640 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-root\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162702 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-tls\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162725 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-sys\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162789 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-sys\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162823 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-wtmp\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162869 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1fc34a-ac92-4787-9859-2535ff564f73-root\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.162977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.162883 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-textfile\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.163511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.163349 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-metrics-client-ca\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.163511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.163419 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.165461 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.165439 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.165575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.165554 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1fc34a-ac92-4787-9859-2535ff564f73-node-exporter-tls\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.227519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.227485 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt925\" (UniqueName: \"kubernetes.io/projected/bc1fc34a-ac92-4787-9859-2535ff564f73-kube-api-access-wt925\") pod \"node-exporter-5gnht\" (UID: \"bc1fc34a-ac92-4787-9859-2535ff564f73\") " pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.293866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.293788 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5gnht" Oct 30 21:23:27.330813 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:27.330770 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1fc34a_ac92_4787_9859_2535ff564f73.slice/crio-2210f555a5d19b89254d6d7be3b3cf4fc48861554d74c01495429324a1d2f3bd WatchSource:0}: Error finding container 2210f555a5d19b89254d6d7be3b3cf4fc48861554d74c01495429324a1d2f3bd: Status 404 returned error can't find the container with id 2210f555a5d19b89254d6d7be3b3cf4fc48861554d74c01495429324a1d2f3bd Oct 30 21:23:27.638099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:27.638068 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gnht" event={"ID":"bc1fc34a-ac92-4787-9859-2535ff564f73","Type":"ContainerStarted","Data":"2210f555a5d19b89254d6d7be3b3cf4fc48861554d74c01495429324a1d2f3bd"} Oct 30 21:23:29.583311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:29.583238 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fddbbf884-7gkgc" Oct 30 21:23:29.646657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:29.646594 4525 generic.go:356] "Generic (PLEG): container finished" podID="bc1fc34a-ac92-4787-9859-2535ff564f73" containerID="0868ea9af2d634d38b4b87ee4e9292fde53a29ff6cbd91fd2b7e21cc4299ec52" exitCode=0 Oct 30 21:23:29.646657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:29.646637 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gnht" event={"ID":"bc1fc34a-ac92-4787-9859-2535ff564f73","Type":"ContainerDied","Data":"0868ea9af2d634d38b4b87ee4e9292fde53a29ff6cbd91fd2b7e21cc4299ec52"} Oct 30 21:23:30.652338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:30.652308 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gnht" event={"ID":"bc1fc34a-ac92-4787-9859-2535ff564f73","Type":"ContainerStarted","Data":"41eb3b6d2c90df0acdaa27a60c2af8373aa3f717a289f5cb72e2adffec5ba1c0"} Oct 30 21:23:30.652734 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:30.652360 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gnht" event={"ID":"bc1fc34a-ac92-4787-9859-2535ff564f73","Type":"ContainerStarted","Data":"a750dd4954d6d161eb89fcb91131ebba0bd772ed7db7759595b5404c37b08f99"} Oct 30 21:23:30.779736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:30.779580 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5gnht" podStartSLOduration=2.818923372 podStartE2EDuration="4.779562384s" totalImagesPullingTime="1.862669581s" totalInitContainerRuntime="97.969431ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:30.77936953 +0000 UTC m=+35.453416067" watchObservedRunningTime="2026-10-30 21:23:30.779562384 +0000 UTC m=+35.453608919" Oct 30 21:23:35.405220 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:23:35.405184 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:23:35.989336 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:35.989306 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ckh6t_5686b209-4171-4cc7-8f73-b128ca2d068e/serve-healthcheck-canary/0.log" Oct 30 21:23:38.976032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:38.975991 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:23:38.976514 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:38.976257 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" podUID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" containerName="insights-gathering" containerID="cri-o://a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914" gracePeriod=30 Oct 30 21:23:39.236896 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.236827 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:23:39.346495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346462 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") pod \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " Oct 30 21:23:39.346700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346502 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") pod \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " Oct 30 21:23:39.346700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346529 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") pod \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " Oct 30 21:23:39.346700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346689 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") pod \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\" (UID: \"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300\") " Oct 30 21:23:39.346916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346894 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle" pod "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" (UID: "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:39.346979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346939 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle" pod "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" (UID: "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:39.347021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.346973 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path" pod "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" (UID: "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:39.349078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.349055 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch" pod "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" (UID: "eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300"). InnerVolumeSpecName "kube-api-access-g87ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:39.447358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.447325 4525 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-trusted-ca-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:39.447358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.447353 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g87ch\" (UniqueName: \"kubernetes.io/projected/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-kube-api-access-g87ch\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:39.447358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.447365 4525 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-archives-path\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:39.447591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.447373 4525 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300-service-ca-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:39.683072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.683033 4525 generic.go:356] "Generic (PLEG): container finished" podID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" containerID="a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914" exitCode=2 Oct 30 21:23:39.683232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.683081 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" Oct 30 21:23:39.683232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.683118 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" event={"ID":"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300","Type":"ContainerDied","Data":"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914"} Oct 30 21:23:39.683232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.683178 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6bvq9-2cdd6" event={"ID":"eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300","Type":"ContainerDied","Data":"a4bd4f0e8bcbcc1ff37d898bf0171a42c4f3f69ae79032e991e09c6fb0b7452f"} Oct 30 21:23:39.683232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.683200 4525 scope.go:122] "RemoveContainer" containerID="a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914" Oct 30 21:23:39.691215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.691194 4525 scope.go:122] "RemoveContainer" containerID="a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914" Oct 30 21:23:39.691516 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:23:39.691478 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914\": container with ID starting with a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914 not found: ID does not exist" containerID="a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914" Oct 30 21:23:39.691564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.691528 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914"} err="failed to get container status \"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914\": rpc error: code = NotFound desc = could not find container \"a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914\": container with ID starting with a4cdd180edc413195c4d393340e641d4813cd7861f2c568f4ca7314a31584914 not found: ID does not exist" Oct 30 21:23:39.847086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.847054 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:23:39.897074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:39.897043 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-6bvq9-2cdd6"] Oct 30 21:23:41.470663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:41.470634 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" path="/var/lib/kubelet/pods/eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300/volumes" Oct 30 21:23:44.734805 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.734773 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:23:44.735290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.735142 4525 cpu_manager.go:414] "containerMap: removing container" podUID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" containerName="insights-gathering" Oct 30 21:23:44.735290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.735160 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" containerName="insights-gathering" Oct 30 21:23:44.735290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.735218 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec5d8d2-70c3-4a6a-a0ba-4f17f3c01300" containerName="insights-gathering" Oct 30 21:23:44.737682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.737663 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.792388 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.792355 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:23:44.792388 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.792355 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:23:44.842010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.841983 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:23:44.842201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.842060 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k65vr\"" Oct 30 21:23:44.842201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.842101 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:23:44.842201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.842123 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:23:44.887689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887652 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887690 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvnw\" (UniqueName: \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887728 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887793 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887863 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887906 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.887933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887926 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.888032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.887951 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.892488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.892473 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:23:44.942752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.942721 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:23:44.942915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.942809 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:23:44.943587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.943571 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:23:44.989322 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989241 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989322 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989284 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989406 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989435 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989470 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989652 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvnw\" (UniqueName: \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989711 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989744 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.989815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.989764 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.990412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.990389 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.990506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.990429 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.990506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.990450 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.990506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.990467 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.991999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.991981 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:44.991999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:44.991993 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:45.051456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:45.051423 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvnw\" (UniqueName: \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\") pod \"console-59c555b9f4-kmhk8\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:45.347470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:45.347386 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:45.718758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:45.718727 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:23:45.750029 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:45.750005 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499c978c_4d49_476f_9920_af5a1db54d1e.slice/crio-543374518fd1fcf9cda877aa37f16f3f97d9a1382c21da7d7324c98597588bdf WatchSource:0}: Error finding container 543374518fd1fcf9cda877aa37f16f3f97d9a1382c21da7d7324c98597588bdf: Status 404 returned error can't find the container with id 543374518fd1fcf9cda877aa37f16f3f97d9a1382c21da7d7324c98597588bdf Oct 30 21:23:46.707869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:46.707830 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c555b9f4-kmhk8" event={"ID":"499c978c-4d49-476f-9920-af5a1db54d1e","Type":"ContainerStarted","Data":"543374518fd1fcf9cda877aa37f16f3f97d9a1382c21da7d7324c98597588bdf"} Oct 30 21:23:49.719854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:49.719804 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c555b9f4-kmhk8" event={"ID":"499c978c-4d49-476f-9920-af5a1db54d1e","Type":"ContainerStarted","Data":"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19"} Oct 30 21:23:49.839348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:49.839297 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59c555b9f4-kmhk8" podStartSLOduration=2.894885188 podStartE2EDuration="5.839281383s" totalImagesPullingTime="2.944396195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:49.838566608 +0000 UTC m=+54.512613156" watchObservedRunningTime="2026-10-30 21:23:49.839281383 +0000 UTC m=+54.513327914" Oct 30 21:23:55.348548 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:55.348504 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:55.349204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:55.348582 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:55.354050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:55.354026 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:55.742164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:55.742140 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:23:56.163679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.163642 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f8777668-2kc55"] Oct 30 21:23:56.166493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.166476 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.232854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.232818 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f8777668-2kc55"] Oct 30 21:23:56.272383 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272344 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7xm\" (UniqueName: \"kubernetes.io/projected/b4d86910-b2d4-4888-8f18-f07569d96377-kube-api-access-th7xm\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272388 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-service-ca\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272477 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-console-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272505 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d86910-b2d4-4888-8f18-f07569d96377-tmp\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272542 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272644 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-oauth-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272680 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-trusted-ca-bundle\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.272752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.272701 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-oauth-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.373750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373703 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-oauth-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.373750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373758 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-trusted-ca-bundle\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374268 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373886 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-oauth-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374268 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373936 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7xm\" (UniqueName: \"kubernetes.io/projected/b4d86910-b2d4-4888-8f18-f07569d96377-kube-api-access-th7xm\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374268 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373966 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-service-ca\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374268 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.373999 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-console-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374268 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374190 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d86910-b2d4-4888-8f18-f07569d96377-tmp\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374276 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374661 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d86910-b2d4-4888-8f18-f07569d96377-tmp\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374874 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-trusted-ca-bundle\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374928 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-console-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.374968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374939 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-service-ca\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.375093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.374961 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4d86910-b2d4-4888-8f18-f07569d96377-oauth-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.376433 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.376409 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-oauth-config\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.376910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.376894 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d86910-b2d4-4888-8f18-f07569d96377-console-serving-cert\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.436166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.436099 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7xm\" (UniqueName: \"kubernetes.io/projected/b4d86910-b2d4-4888-8f18-f07569d96377-kube-api-access-th7xm\") pod \"console-66f8777668-2kc55\" (UID: \"b4d86910-b2d4-4888-8f18-f07569d96377\") " pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.476225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.476190 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:23:56.847393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:56.847321 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f8777668-2kc55"] Oct 30 21:23:56.850249 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:23:56.850218 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d86910_b2d4_4888_8f18_f07569d96377.slice/crio-202f4fb9d73e01b693e1878f711d09ffd31e2a1e755011dc5e07ff42fcca2a5a WatchSource:0}: Error finding container 202f4fb9d73e01b693e1878f711d09ffd31e2a1e755011dc5e07ff42fcca2a5a: Status 404 returned error can't find the container with id 202f4fb9d73e01b693e1878f711d09ffd31e2a1e755011dc5e07ff42fcca2a5a Oct 30 21:23:57.744992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:57.744956 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f8777668-2kc55" event={"ID":"b4d86910-b2d4-4888-8f18-f07569d96377","Type":"ContainerStarted","Data":"63358c949d6648a2ff84ed31f21d354a59d4d72bad33d1bfcd2b1b48ab5304b7"} Oct 30 21:23:57.745403 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:57.745003 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f8777668-2kc55" event={"ID":"b4d86910-b2d4-4888-8f18-f07569d96377","Type":"ContainerStarted","Data":"202f4fb9d73e01b693e1878f711d09ffd31e2a1e755011dc5e07ff42fcca2a5a"} Oct 30 21:23:57.899418 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:23:57.899360 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66f8777668-2kc55" podStartSLOduration=1.8993432989999999 podStartE2EDuration="1.899343299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:57.898700313 +0000 UTC m=+62.572746851" watchObservedRunningTime="2026-10-30 21:23:57.899343299 +0000 UTC m=+62.573389834" Oct 30 21:24:04.951254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:04.951210 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:04.953881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:04.953841 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.008301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.008270 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:24:05.008301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.008306 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:24:05.008519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.008320 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:24:05.038521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038478 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-web-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038520 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038549 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038641 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038716 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038737 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038754 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6sz\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-kube-api-access-vw6sz\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038782 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038870 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-out\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038892 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038911 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.038941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.038928 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.039098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.039010 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.058137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.058111 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:24:05.058218 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.058181 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:24:05.058300 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.058281 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:24:05.058300 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.058295 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:24:05.058814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.058798 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bdcrp\"" Oct 30 21:24:05.109334 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.109292 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:24:05.139871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.139839 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.139884 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-web-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.139913 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.139953 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.139980 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140124 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140171 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140194 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6sz\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-kube-api-access-vw6sz\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140234 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140276 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-out\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140300 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140325 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140346 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.140655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.140452 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.141045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.141021 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.143263 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.143242 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.143471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.143450 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.143675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.143647 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.143818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.143800 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.143970 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.143952 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-volume\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.144199 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.144177 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.144289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.144180 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-web-config\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.144328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.144288 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db755ab9-3225-430e-b8c0-1fa42bf52a0b-config-out\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.145560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.145543 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db755ab9-3225-430e-b8c0-1fa42bf52a0b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.158961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.158939 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:24:05.159934 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.159913 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:05.161289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.161270 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db755ab9-3225-430e-b8c0-1fa42bf52a0b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.203163 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.203101 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6sz\" (UniqueName: \"kubernetes.io/projected/db755ab9-3225-430e-b8c0-1fa42bf52a0b-kube-api-access-vw6sz\") pod \"alertmanager-main-0\" (UID: \"db755ab9-3225-430e-b8c0-1fa42bf52a0b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.271219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.271181 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:24:05.394513 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:24:05.394476 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:24:05.703724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.703696 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:24:05.707059 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:24:05.707025 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb755ab9_3225_430e_b8c0_1fa42bf52a0b.slice/crio-ee3fc55627556640323f341aadd685bfd27ad7100131a9542bd39f7a4cb84515 WatchSource:0}: Error finding container ee3fc55627556640323f341aadd685bfd27ad7100131a9542bd39f7a4cb84515: Status 404 returned error can't find the container with id ee3fc55627556640323f341aadd685bfd27ad7100131a9542bd39f7a4cb84515 Oct 30 21:24:05.776223 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:05.776186 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"ee3fc55627556640323f341aadd685bfd27ad7100131a9542bd39f7a4cb84515"} Oct 30 21:24:06.476372 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:06.476338 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:24:06.476821 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:06.476408 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:24:06.481079 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:06.481057 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:24:06.785322 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:06.785251 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f8777668-2kc55" Oct 30 21:24:06.941871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:06.941838 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:24:07.784808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:07.784770 4525 generic.go:356] "Generic (PLEG): container finished" podID="db755ab9-3225-430e-b8c0-1fa42bf52a0b" containerID="f60d0e7d1a1a9c7963d415148f6ca5aa1a015b59301f00894d3168a25e35acd5" exitCode=0 Oct 30 21:24:07.785226 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:07.784863 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerDied","Data":"f60d0e7d1a1a9c7963d415148f6ca5aa1a015b59301f00894d3168a25e35acd5"} Oct 30 21:24:09.793789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:09.793762 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"94d59a466b4212bcfd9d0f0781d695657d77e297b6de4f00e843204c32814afc"} Oct 30 21:24:09.794101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:09.793853 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"0ca27b58461f26d91ea286b553a38429660611092aed89003f3fb22548ffabc8"} Oct 30 21:24:10.801531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:10.801486 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"05c4075f11ecec7e2b5bc6d67087b12dec5878a1aa4c672b0ad222a9c50bf0f6"} Oct 30 21:24:10.801926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:10.801564 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"4ccd62a40eced5481ba4a41812f59377cf8fe76ddb683178e65ad04d03309bdd"} Oct 30 21:24:10.801926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:10.801591 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"4c1dfa036539fbff1a98749d4653128313cb2fd6c3e328c064cc4314c375afc6"} Oct 30 21:24:11.808321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:11.808288 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"db755ab9-3225-430e-b8c0-1fa42bf52a0b","Type":"ContainerStarted","Data":"14927035f4f28c9cfe494b5ee4561996472f76446cb2b1c27cea5d6b634fc99c"} Oct 30 21:24:12.087195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:12.087083 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.640948404 podStartE2EDuration="8.087070753s" totalImagesPullingTime="4.431962166s" totalInitContainerRuntime="14.160183ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:12.086895201 +0000 UTC m=+76.760941738" watchObservedRunningTime="2026-10-30 21:24:12.087070753 +0000 UTC m=+76.761117289" Oct 30 21:24:31.964287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:31.964220 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59c555b9f4-kmhk8" podUID="499c978c-4d49-476f-9920-af5a1db54d1e" containerName="console" containerID="cri-o://dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19" gracePeriod=15 Oct 30 21:24:32.205598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.205574 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:24:32.243706 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243597 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243706 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243673 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243717 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243743 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243797 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243829 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243857 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\" (UniqueName: \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.243937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.243886 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\") pod \"499c978c-4d49-476f-9920-af5a1db54d1e\" (UID: \"499c978c-4d49-476f-9920-af5a1db54d1e\") " Oct 30 21:24:32.244248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.244037 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:32.244306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.244232 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:32.244306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.244245 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:32.244306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.244259 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:32.244466 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.244444 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:32.246245 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.246219 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "kube-api-access-lkvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:32.246564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.246531 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:32.246682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.246542 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert" pod "499c978c-4d49-476f-9920-af5a1db54d1e" (UID: "499c978c-4d49-476f-9920-af5a1db54d1e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:32.344416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344384 4525 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-console-config\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344414 4525 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-serving-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344424 4525 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/499c978c-4d49-476f-9920-af5a1db54d1e-console-oauth-config\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344433 4525 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-oauth-serving-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344441 4525 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-trusted-ca-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344450 4525 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/499c978c-4d49-476f-9920-af5a1db54d1e-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344458 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvnw\" (UniqueName: \"kubernetes.io/projected/499c978c-4d49-476f-9920-af5a1db54d1e-kube-api-access-lkvnw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.344714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.344470 4525 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/499c978c-4d49-476f-9920-af5a1db54d1e-tmp\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:32.881804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.881771 4525 generic.go:356] "Generic (PLEG): container finished" podID="499c978c-4d49-476f-9920-af5a1db54d1e" containerID="dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19" exitCode=0 Oct 30 21:24:32.881972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.881822 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c555b9f4-kmhk8" Oct 30 21:24:32.881972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.881833 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c555b9f4-kmhk8" event={"ID":"499c978c-4d49-476f-9920-af5a1db54d1e","Type":"ContainerDied","Data":"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19"} Oct 30 21:24:32.881972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.881872 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c555b9f4-kmhk8" event={"ID":"499c978c-4d49-476f-9920-af5a1db54d1e","Type":"ContainerDied","Data":"543374518fd1fcf9cda877aa37f16f3f97d9a1382c21da7d7324c98597588bdf"} Oct 30 21:24:32.881972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.881890 4525 scope.go:122] "RemoveContainer" containerID="dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19" Oct 30 21:24:32.891146 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.891127 4525 scope.go:122] "RemoveContainer" containerID="dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19" Oct 30 21:24:32.891409 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:24:32.891390 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19\": container with ID starting with dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19 not found: ID does not exist" containerID="dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19" Oct 30 21:24:32.891462 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:32.891419 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19"} err="failed to get container status \"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19\": rpc error: code = NotFound desc = could not find container \"dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19\": container with ID starting with dd099791fea8e8735ddfa1ca746b1d57c07ab165250d0f1ab9e4ab39d388ab19 not found: ID does not exist" Oct 30 21:24:33.045900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:33.045867 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:24:33.096055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:33.096021 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c555b9f4-kmhk8"] Oct 30 21:24:33.470563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:24:33.470529 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499c978c-4d49-476f-9920-af5a1db54d1e" path="/var/lib/kubelet/pods/499c978c-4d49-476f-9920-af5a1db54d1e/volumes" Oct 30 21:24:35.395168 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:24:35.395131 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:25:02.981939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:02.981907 4525 generic.go:356] "Generic (PLEG): container finished" podID="43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" containerID="25466fae2ef76d4cd63be3955d8c45837b6212cd3770733e5a781206f4c0637b" exitCode=0 Oct 30 21:25:02.982305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:02.981971 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" event={"ID":"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1","Type":"ContainerDied","Data":"25466fae2ef76d4cd63be3955d8c45837b6212cd3770733e5a781206f4c0637b"} Oct 30 21:25:04.119028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.119005 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:25:04.171378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171341 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\") pod \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " Oct 30 21:25:04.171568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171416 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\" (UniqueName: \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\") pod \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " Oct 30 21:25:04.171568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171446 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\") pod \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " Oct 30 21:25:04.171568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171468 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\") pod \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\" (UID: \"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1\") " Oct 30 21:25:04.171794 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171766 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path" pod "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" (UID: "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:04.171928 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171890 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle" pod "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" (UID: "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:04.171928 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.171909 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle" pod "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" (UID: "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:04.173827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.173805 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd" pod "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" (UID: "43b523b1-c52d-416d-8ddc-ac5d51b0b5b1"). InnerVolumeSpecName "kube-api-access-45qsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:04.272478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.272385 4525 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-service-ca-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:04.272478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.272420 4525 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-trusted-ca-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:04.272478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.272432 4525 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-archives-path\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:04.272478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.272444 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qsd\" (UniqueName: \"kubernetes.io/projected/43b523b1-c52d-416d-8ddc-ac5d51b0b5b1-kube-api-access-45qsd\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:04.989796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.989760 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" Oct 30 21:25:04.989987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.989756 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-w9wjt-4qhsz" event={"ID":"43b523b1-c52d-416d-8ddc-ac5d51b0b5b1","Type":"ContainerDied","Data":"64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc"} Oct 30 21:25:04.989987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:25:04.989894 4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fdf8d8cb0931d3f60baef1a966c4d5ea4886c8965ec19d72ff5da8dab78dbc" Oct 30 21:25:05.395731 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:25:05.395699 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:25:35.395143 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:25:35.395061 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:26:05.395380 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:26:05.395336 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:26:14.957331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957291 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v"] Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957687 4525 cpu_manager.go:414] "containerMap: removing container" podUID="499c978c-4d49-476f-9920-af5a1db54d1e" containerName="console" Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957704 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499c978c-4d49-476f-9920-af5a1db54d1e" containerName="console" Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957736 4525 cpu_manager.go:414] "containerMap: removing container" podUID="43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" containerName="insights-gathering" Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957744 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" containerName="insights-gathering" Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957828 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="499c978c-4d49-476f-9920-af5a1db54d1e" containerName="console" Oct 30 21:26:14.957955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.957843 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b523b1-c52d-416d-8ddc-ac5d51b0b5b1" containerName="insights-gathering" Oct 30 21:26:14.960927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.960904 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:14.974321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.974288 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzzk\" (UniqueName: \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:14.974433 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.974373 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:14.974433 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:14.974411 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.067233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.067197 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9d662\"" Oct 30 21:26:15.067400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.067258 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:26:15.067400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.067321 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:26:15.068324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.068304 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v"] Oct 30 21:26:15.075753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.075734 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzzk\" (UniqueName: \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.075801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.075769 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.075838 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.075800 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.076196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.076178 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.076230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.076192 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.137369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.137339 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzzk\" (UniqueName: \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.262493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.262414 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:26:15.265574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.265558 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.276843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276823 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.276951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276899 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.276951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276905 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:15.276951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276936 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.277073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276964 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.277073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.276995 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.277131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.277073 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.277131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.277107 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbrb\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.319464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319428 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:26:15.319464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319480 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:26:15.319749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319451 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:26:15.319749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319428 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:26:15.319749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319548 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:26:15.319749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319679 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:26:15.319959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.319805 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9kbqs\"" Oct 30 21:26:15.329134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.329057 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:26:15.378182 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378146 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378192 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378230 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbrb\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378275 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378308 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378330 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378573 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378355 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.378867 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.378837 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.380749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.380720 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.380909 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.380880 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.381127 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.381113 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.381173 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.381159 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.439879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.439839 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.440021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.439980 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbrb\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\") pod \"istiod-openshift-gateway-7876bfcb74-qcnx4\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.576822 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.576729 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:15.649816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.649574 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v"] Oct 30 21:26:15.653668 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:26:15.653641 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8d199_fbf3_46e0_85dd_cfddb480509f.slice/crio-3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b WatchSource:0}: Error finding container 3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b: Status 404 returned error can't find the container with id 3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b Oct 30 21:26:15.953666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:15.953633 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:26:15.956474 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:26:15.956446 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf852fe6d_f39b_46c0_8610_7faf8383ced0.slice/crio-ab2703b8262ac00fbaeafdbcad25352418c5f16187d623ecb1a5840fffbc74c8 WatchSource:0}: Error finding container ab2703b8262ac00fbaeafdbcad25352418c5f16187d623ecb1a5840fffbc74c8: Status 404 returned error can't find the container with id ab2703b8262ac00fbaeafdbcad25352418c5f16187d623ecb1a5840fffbc74c8 Oct 30 21:26:16.223705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:16.223596 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" event={"ID":"f852fe6d-f39b-46c0-8610-7faf8383ced0","Type":"ContainerStarted","Data":"ab2703b8262ac00fbaeafdbcad25352418c5f16187d623ecb1a5840fffbc74c8"} Oct 30 21:26:16.224825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:16.224779 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerStarted","Data":"3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b"} Oct 30 21:26:19.980514 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:19.980478 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:26:19.980936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:19.980544 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:26:20.249053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:20.248957 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" event={"ID":"f852fe6d-f39b-46c0-8610-7faf8383ced0","Type":"ContainerStarted","Data":"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69"} Oct 30 21:26:20.249336 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:20.249275 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:20.471260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:20.471199 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" podStartSLOduration=1.4497819299999999 podStartE2EDuration="5.470999909s" totalImagesPullingTime="4.021217979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:20.419780034 +0000 UTC m=+205.093826593" watchObservedRunningTime="2026-10-30 21:26:20.470999909 +0000 UTC m=+205.145046446" Oct 30 21:26:21.256162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:21.256113 4525 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-qcnx4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:26:21.256685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:21.256180 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:26:22.258841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:22.258804 4525 generic.go:356] "Generic (PLEG): container finished" podID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerID="4fbce8aaff01038b2be2a96ce386804408045c9843d08012211c3f94f7bd15fd" exitCode=0 Oct 30 21:26:22.258841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:22.258841 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerDied","Data":"4fbce8aaff01038b2be2a96ce386804408045c9843d08012211c3f94f7bd15fd"} Oct 30 21:26:24.255271 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:24.255242 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:26:24.268435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:24.268400 4525 generic.go:356] "Generic (PLEG): container finished" podID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerID="20a67071f1e3ab3c3f88393a3b6908e778fe5200eb3a5ea006a3c648c7fee363" exitCode=0 Oct 30 21:26:24.268568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:24.268447 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerDied","Data":"20a67071f1e3ab3c3f88393a3b6908e778fe5200eb3a5ea006a3c648c7fee363"} Oct 30 21:26:31.296261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:31.296224 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerStarted","Data":"8c0b5f3b8f5245a8140ae730ea71320d54bb8178ba2346866a763238fe5fb9d1"} Oct 30 21:26:31.414946 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:31.414892 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" podStartSLOduration=3.563111622 podStartE2EDuration="17.414876694s" totalImagesPullingTime="13.851765072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:31.413884223 +0000 UTC m=+216.087930790" watchObservedRunningTime="2026-10-30 21:26:31.414876694 +0000 UTC m=+216.088923230" Oct 30 21:26:32.302064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:32.302024 4525 generic.go:356] "Generic (PLEG): container finished" podID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerID="8c0b5f3b8f5245a8140ae730ea71320d54bb8178ba2346866a763238fe5fb9d1" exitCode=0 Oct 30 21:26:32.302532 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:32.302102 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerDied","Data":"8c0b5f3b8f5245a8140ae730ea71320d54bb8178ba2346866a763238fe5fb9d1"} Oct 30 21:26:33.424834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.424808 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:33.528159 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.528125 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\") pod \"79e8d199-fbf3-46e0-85dd-cfddb480509f\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " Oct 30 21:26:33.528363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.528174 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\" (UniqueName: \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\") pod \"79e8d199-fbf3-46e0-85dd-cfddb480509f\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " Oct 30 21:26:33.528363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.528191 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\") pod \"79e8d199-fbf3-46e0-85dd-cfddb480509f\" (UID: \"79e8d199-fbf3-46e0-85dd-cfddb480509f\") " Oct 30 21:26:33.528801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.528775 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle" pod "79e8d199-fbf3-46e0-85dd-cfddb480509f" (UID: "79e8d199-fbf3-46e0-85dd-cfddb480509f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:33.530527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.530504 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk" pod "79e8d199-fbf3-46e0-85dd-cfddb480509f" (UID: "79e8d199-fbf3-46e0-85dd-cfddb480509f"). InnerVolumeSpecName "kube-api-access-srzzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:33.532472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.532452 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util" pod "79e8d199-fbf3-46e0-85dd-cfddb480509f" (UID: "79e8d199-fbf3-46e0-85dd-cfddb480509f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:33.629020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.628995 4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-util\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:33.629177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.629023 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzzk\" (UniqueName: \"kubernetes.io/projected/79e8d199-fbf3-46e0-85dd-cfddb480509f-kube-api-access-srzzk\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:33.629177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:33.629056 4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79e8d199-fbf3-46e0-85dd-cfddb480509f-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:34.311210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:34.311172 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" event={"ID":"79e8d199-fbf3-46e0-85dd-cfddb480509f","Type":"ContainerDied","Data":"3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b"} Oct 30 21:26:34.311210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:34.311193 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71z9s4v" Oct 30 21:26:34.311210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:26:34.311218 4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e43cc571947747e4debf6cb972faf1f46e6a5165288fd4379ccaa556e75bb8b" Oct 30 21:26:35.394810 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:26:35.394777 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:27:05.395224 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:27:05.395185 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:27:14.050747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.050714 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw"] Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.050969 4525 cpu_manager.go:414] "containerMap: removing container" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="pull" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.050980 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="pull" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.050997 4525 cpu_manager.go:414] "containerMap: removing container" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="util" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.051004 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="util" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.051013 4525 cpu_manager.go:414] "containerMap: removing container" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="extract" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.051020 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="extract" Oct 30 21:27:14.051142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.051070 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e8d199-fbf3-46e0-85dd-cfddb480509f" containerName="extract" Oct 30 21:27:14.053062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.053040 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.105670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.105641 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:27:14.143645 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.143579 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2nk\" (UniqueName: \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.143816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.143726 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.143816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.143764 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.155658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.155601 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:27:14.155805 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.155698 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rjl5v\"" Oct 30 21:27:14.156704 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.156681 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw"] Oct 30 21:27:14.244174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.244137 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2nk\" (UniqueName: \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.244381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.244197 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.244381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.244224 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.244646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.244599 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.244704 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.244644 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.306452 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.306378 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2nk\" (UniqueName: \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.364512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.364479 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:14.736427 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:14.736387 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw"] Oct 30 21:27:14.771093 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:27:14.771064 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6ce60a_624b_4d19_b4de_74207d0d1b78.slice/crio-b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5 WatchSource:0}: Error finding container b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5: Status 404 returned error can't find the container with id b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5 Oct 30 21:27:15.447360 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:15.447326 4525 generic.go:356] "Generic (PLEG): container finished" podID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerID="ecfd3bb2a9d8f11d8bfa57ba9fc5a5cd18a9b60f089d10339b8fd31b2a84ae22" exitCode=0 Oct 30 21:27:15.447791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:15.447408 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" event={"ID":"4f6ce60a-624b-4d19-b4de-74207d0d1b78","Type":"ContainerDied","Data":"ecfd3bb2a9d8f11d8bfa57ba9fc5a5cd18a9b60f089d10339b8fd31b2a84ae22"} Oct 30 21:27:15.447791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:15.447447 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" event={"ID":"4f6ce60a-624b-4d19-b4de-74207d0d1b78","Type":"ContainerStarted","Data":"b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5"} Oct 30 21:27:17.456377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:17.456338 4525 generic.go:356] "Generic (PLEG): container finished" podID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerID="3e9271b35cfbcbaf2b562409de8b08ecc4f0c3c55e475c115f3622829ebcb5f6" exitCode=0 Oct 30 21:27:17.456377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:17.456379 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" event={"ID":"4f6ce60a-624b-4d19-b4de-74207d0d1b78","Type":"ContainerDied","Data":"3e9271b35cfbcbaf2b562409de8b08ecc4f0c3c55e475c115f3622829ebcb5f6"} Oct 30 21:27:18.231860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.231778 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb"] Oct 30 21:27:18.234224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.234203 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.307752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307715 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:27:18.307935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307769 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:27:18.307935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307835 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:27:18.307935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307836 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9g9fn\"" Oct 30 21:27:18.307935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307864 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:27:18.307935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.307864 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:27:18.367516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.367475 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb"] Oct 30 21:27:18.382670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382640 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.382841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382680 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.382841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382748 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.382841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382778 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.382841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382804 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.382971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.382911 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjnn\" (UniqueName: \"kubernetes.io/projected/d674c509-d414-417f-aa58-3e7e5badb469-kube-api-access-szjnn\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.462310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.462278 4525 generic.go:356] "Generic (PLEG): container finished" podID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerID="beef089fa846874af2a773f20c66ab8cf18149e83e5f6c468c4529b403e56b88" exitCode=0 Oct 30 21:27:18.462720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.462337 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" event={"ID":"4f6ce60a-624b-4d19-b4de-74207d0d1b78","Type":"ContainerDied","Data":"beef089fa846874af2a773f20c66ab8cf18149e83e5f6c468c4529b403e56b88"} Oct 30 21:27:18.483827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.483765 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.483827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.483804 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.483951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.483834 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.483951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.483866 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.483951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.483904 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.484139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.484113 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjnn\" (UniqueName: \"kubernetes.io/projected/d674c509-d414-417f-aa58-3e7e5badb469-kube-api-access-szjnn\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.484258 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.484234 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-manifests\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.486474 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.486450 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-metrics-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.486597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.486584 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-webhook-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.486708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.486688 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d674c509-d414-417f-aa58-3e7e5badb469-tmp\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.486901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.486886 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d674c509-d414-417f-aa58-3e7e5badb469-apiservice-cert\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.545688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.545659 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjnn\" (UniqueName: \"kubernetes.io/projected/d674c509-d414-417f-aa58-3e7e5badb469-kube-api-access-szjnn\") pod \"opendatahub-operator-controller-manager-568bcd9cdd-b2pjb\" (UID: \"d674c509-d414-417f-aa58-3e7e5badb469\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:18.545874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:18.545797 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:19.095431 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.095401 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb"] Oct 30 21:27:19.096458 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:27:19.096434 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd674c509_d414_417f_aa58_3e7e5badb469.slice/crio-3088ff92918f71b1bea3d093d8aae2d210a5a891360b26d802cd1fa9e2ad73bd WatchSource:0}: Error finding container 3088ff92918f71b1bea3d093d8aae2d210a5a891360b26d802cd1fa9e2ad73bd: Status 404 returned error can't find the container with id 3088ff92918f71b1bea3d093d8aae2d210a5a891360b26d802cd1fa9e2ad73bd Oct 30 21:27:19.470866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.470823 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" event={"ID":"d674c509-d414-417f-aa58-3e7e5badb469","Type":"ContainerStarted","Data":"3088ff92918f71b1bea3d093d8aae2d210a5a891360b26d802cd1fa9e2ad73bd"} Oct 30 21:27:19.585382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.585356 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:19.591028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.591008 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\" (UniqueName: \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\") pod \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " Oct 30 21:27:19.591088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.591063 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\") pod \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " Oct 30 21:27:19.591122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.591086 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\") pod \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\" (UID: \"4f6ce60a-624b-4d19-b4de-74207d0d1b78\") " Oct 30 21:27:19.592136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.592091 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle" pod "4f6ce60a-624b-4d19-b4de-74207d0d1b78" (UID: "4f6ce60a-624b-4d19-b4de-74207d0d1b78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:19.597868 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.597818 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk" pod "4f6ce60a-624b-4d19-b4de-74207d0d1b78" (UID: "4f6ce60a-624b-4d19-b4de-74207d0d1b78"). InnerVolumeSpecName "kube-api-access-xs2nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:19.599473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.599442 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util" pod "4f6ce60a-624b-4d19-b4de-74207d0d1b78" (UID: "4f6ce60a-624b-4d19-b4de-74207d0d1b78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:19.691815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.691781 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2nk\" (UniqueName: \"kubernetes.io/projected/4f6ce60a-624b-4d19-b4de-74207d0d1b78-kube-api-access-xs2nk\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:19.691815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.691810 4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-bundle\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:19.691815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:19.691821 4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f6ce60a-624b-4d19-b4de-74207d0d1b78-util\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:20.475381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:20.475351 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" Oct 30 21:27:20.475901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:20.475345 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cs8lrw" event={"ID":"4f6ce60a-624b-4d19-b4de-74207d0d1b78","Type":"ContainerDied","Data":"b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5"} Oct 30 21:27:20.475901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:20.475507 4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b373b9e9ef006cc2b1935b4140e1f73a66c2fb224155064579bc372319606bc5" Oct 30 21:27:22.484813 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:22.484774 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" event={"ID":"d674c509-d414-417f-aa58-3e7e5badb469","Type":"ContainerStarted","Data":"51286d1721383aacb9a5724563af765ffa8974a1a5d0bb3326f721ac5691e569"} Oct 30 21:27:23.490019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:23.489985 4525 generic.go:356] "Generic (PLEG): container finished" podID="d674c509-d414-417f-aa58-3e7e5badb469" containerID="51286d1721383aacb9a5724563af765ffa8974a1a5d0bb3326f721ac5691e569" exitCode=0 Oct 30 21:27:23.490423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:23.490078 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" event={"ID":"d674c509-d414-417f-aa58-3e7e5badb469","Type":"ContainerDied","Data":"51286d1721383aacb9a5724563af765ffa8974a1a5d0bb3326f721ac5691e569"} Oct 30 21:27:24.495454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:24.495367 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" event={"ID":"d674c509-d414-417f-aa58-3e7e5badb469","Type":"ContainerStarted","Data":"238f60293d4582443ed420e34a6f6339c204de6f9550c48015e844be8ddb2380"} Oct 30 21:27:24.495855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:24.495505 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:24.751206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:24.748125 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" podStartSLOduration=2.212279831 podStartE2EDuration="6.748106257s" totalImagesPullingTime="4.048721804s" totalInitContainerRuntime="487.104622ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:24.696598589 +0000 UTC m=+269.370645129" watchObservedRunningTime="2026-10-30 21:27:24.748106257 +0000 UTC m=+269.422152793" Oct 30 21:27:35.395319 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:27:35.395276 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:27:35.501473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:35.501444 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-568bcd9cdd-b2pjb" Oct 30 21:27:59.342057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342021 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-2j5st"] Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342299 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="extract" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342310 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="extract" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342325 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="util" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342329 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="util" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342343 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="pull" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342349 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="pull" Oct 30 21:27:59.342537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.342389 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6ce60a-624b-4d19-b4de-74207d0d1b78" containerName="extract" Oct 30 21:27:59.349512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.349490 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.406135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.406104 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj29\" (UniqueName: \"kubernetes.io/projected/598b7c39-8df0-4cf2-b736-a2254f9a00e6-kube-api-access-9gj29\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.406306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.406165 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tmp\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.406306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.406280 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tls-certs\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.452802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.452765 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:27:59.452976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.452893 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:27:59.452976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.452971 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gnpm5\"" Oct 30 21:27:59.454017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.453994 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-2j5st"] Oct 30 21:27:59.507340 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.507303 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tls-certs\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.507535 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.507359 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj29\" (UniqueName: \"kubernetes.io/projected/598b7c39-8df0-4cf2-b736-a2254f9a00e6-kube-api-access-9gj29\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.507535 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.507399 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tmp\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.509820 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.509793 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tmp\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.509969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.509946 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598b7c39-8df0-4cf2-b736-a2254f9a00e6-tls-certs\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.568450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.568409 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj29\" (UniqueName: \"kubernetes.io/projected/598b7c39-8df0-4cf2-b736-a2254f9a00e6-kube-api-access-9gj29\") pod \"kube-auth-proxy-6bcd447999-2j5st\" (UID: \"598b7c39-8df0-4cf2-b736-a2254f9a00e6\") " pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:27:59.661549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:27:59.661515 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" Oct 30 21:28:00.039417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:00.039342 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcd447999-2j5st"] Oct 30 21:28:00.042478 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:28:00.042450 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598b7c39_8df0_4cf2_b736_a2254f9a00e6.slice/crio-49fd2410d5e16fe601d85ef40ce75f9f32537188ba37da1e8c69af9ce432877a WatchSource:0}: Error finding container 49fd2410d5e16fe601d85ef40ce75f9f32537188ba37da1e8c69af9ce432877a: Status 404 returned error can't find the container with id 49fd2410d5e16fe601d85ef40ce75f9f32537188ba37da1e8c69af9ce432877a Oct 30 21:28:00.044335 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:00.044320 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:28:00.621745 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:00.621707 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" event={"ID":"598b7c39-8df0-4cf2-b736-a2254f9a00e6","Type":"ContainerStarted","Data":"49fd2410d5e16fe601d85ef40ce75f9f32537188ba37da1e8c69af9ce432877a"} Oct 30 21:28:03.636065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:03.636029 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" event={"ID":"598b7c39-8df0-4cf2-b736-a2254f9a00e6","Type":"ContainerStarted","Data":"7a2bec03b1620e6435362b5ecb216dffe680eae2c5e62e310858338feb8bc403"} Oct 30 21:28:03.776510 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:03.776458 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bcd447999-2j5st" podStartSLOduration=1.9013049610000001 podStartE2EDuration="4.776442752s" totalImagesPullingTime="2.875137791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:03.774505927 +0000 UTC m=+308.448552462" watchObservedRunningTime="2026-10-30 21:28:03.776442752 +0000 UTC m=+308.450489265" Oct 30 21:28:05.369027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.368991 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:28:05.372420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.372387 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.397625 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:05.397579 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:28:05.475057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.475017 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:28:05.475057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.475031 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dpbcr\"" Oct 30 21:28:05.475057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.475017 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:28:05.475057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.475031 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:28:05.476266 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.476249 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:28:05.477523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.477503 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/571e4219-7208-48ea-80dd-f2118d4293d6-config-file\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.477641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.477549 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm9h\" (UniqueName: \"kubernetes.io/projected/571e4219-7208-48ea-80dd-f2118d4293d6-kube-api-access-bqm9h\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.578855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.578822 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/571e4219-7208-48ea-80dd-f2118d4293d6-config-file\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.579043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.578870 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm9h\" (UniqueName: \"kubernetes.io/projected/571e4219-7208-48ea-80dd-f2118d4293d6-kube-api-access-bqm9h\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.579620 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.579581 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/571e4219-7208-48ea-80dd-f2118d4293d6-config-file\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.644978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.644898 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm9h\" (UniqueName: \"kubernetes.io/projected/571e4219-7208-48ea-80dd-f2118d4293d6-kube-api-access-bqm9h\") pod \"limitador-limitador-7c9b87d574-l7jz6\" (UID: \"571e4219-7208-48ea-80dd-f2118d4293d6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:05.670506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.670471 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:28:05.685374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:05.685345 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:06.056586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:06.056559 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:28:06.059057 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:28:06.059026 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571e4219_7208_48ea_80dd_f2118d4293d6.slice/crio-3ba92be975bcd2b621d67924fa3f854b6f926fedae643a841732f1a3c3a10858 WatchSource:0}: Error finding container 3ba92be975bcd2b621d67924fa3f854b6f926fedae643a841732f1a3c3a10858: Status 404 returned error can't find the container with id 3ba92be975bcd2b621d67924fa3f854b6f926fedae643a841732f1a3c3a10858 Oct 30 21:28:06.650624 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:06.650567 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" event={"ID":"571e4219-7208-48ea-80dd-f2118d4293d6","Type":"ContainerStarted","Data":"3ba92be975bcd2b621d67924fa3f854b6f926fedae643a841732f1a3c3a10858"} Oct 30 21:28:09.665371 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:09.665332 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" event={"ID":"571e4219-7208-48ea-80dd-f2118d4293d6","Type":"ContainerStarted","Data":"fabe200ef8f4c68063a2816a5b9d1118fbf1a41054f959c1c22e3b94882d1089"} Oct 30 21:28:09.665824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:09.665492 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:09.782386 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:09.782330 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" podStartSLOduration=1.821064131 podStartE2EDuration="4.782316007s" totalImagesPullingTime="2.961251876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:09.781722741 +0000 UTC m=+314.455769287" watchObservedRunningTime="2026-10-30 21:28:09.782316007 +0000 UTC m=+314.456362570" Oct 30 21:28:20.670823 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:20.670788 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-l7jz6" Oct 30 21:28:35.395876 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:35.395833 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:28:51.328865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.328831 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:28:51.331234 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.331212 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.383974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.383946 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:28:51.383974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.383959 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:28:51.434031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.433985 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:28:51.434031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.433986 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:28:51.434286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.434087 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cl4bb\"" Oct 30 21:28:51.434286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.434182 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:28:51.435301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.435280 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:28:51.455842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.455815 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzg2\" (UniqueName: \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.456011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.455854 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.456011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.455965 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.557369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.557327 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.557590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.557397 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzg2\" (UniqueName: \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.557590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.557434 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.560049 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.560015 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.560269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.560250 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.618799 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.618774 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzg2\" (UniqueName: \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\") pod \"maas-controller-579cf7c6d-rr2sq\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.642782 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.642742 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:51.751507 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.751474 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 21:28:51.753927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.753902 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.816035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.816004 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 21:28:51.858549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.858507 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.858764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.858653 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl2p\" (UniqueName: \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.858764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.858714 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.959474 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.959378 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.959474 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.959448 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl2p\" (UniqueName: \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.959751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.959491 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.962080 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.962048 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:51.962197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:51.962085 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:52.019156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.019137 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:28:52.021724 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:28:52.021700 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65264b5_3775_4cfd_8cb8_10f015d801a0.slice/crio-6def7dc3ae108ff118a3d5ccc583ef83d9065645dd1977a448f94c7b62bfdad7 WatchSource:0}: Error finding container 6def7dc3ae108ff118a3d5ccc583ef83d9065645dd1977a448f94c7b62bfdad7: Status 404 returned error can't find the container with id 6def7dc3ae108ff118a3d5ccc583ef83d9065645dd1977a448f94c7b62bfdad7 Oct 30 21:28:52.024722 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.024702 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl2p\" (UniqueName: \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\") pod \"maas-controller-685458bf9d-bz826\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:52.065652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.065621 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:52.437478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.437457 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 21:28:52.439468 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:28:52.439441 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f863a5_33e2_4314_a6fc_c95d748ab8a9.slice/crio-79ff034ddfdc1de9aa4c6fda417f8411279574707491e453d2a0254689582ef9 WatchSource:0}: Error finding container 79ff034ddfdc1de9aa4c6fda417f8411279574707491e453d2a0254689582ef9: Status 404 returned error can't find the container with id 79ff034ddfdc1de9aa4c6fda417f8411279574707491e453d2a0254689582ef9 Oct 30 21:28:52.834020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.833922 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerStarted","Data":"79ff034ddfdc1de9aa4c6fda417f8411279574707491e453d2a0254689582ef9"} Oct 30 21:28:52.835011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:52.834989 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" event={"ID":"a65264b5-3775-4cfd-8cb8-10f015d801a0","Type":"ContainerStarted","Data":"6def7dc3ae108ff118a3d5ccc583ef83d9065645dd1977a448f94c7b62bfdad7"} Oct 30 21:28:55.849585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:55.849542 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" event={"ID":"a65264b5-3775-4cfd-8cb8-10f015d801a0","Type":"ContainerStarted","Data":"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e"} Oct 30 21:28:55.850066 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:55.849665 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:28:55.851188 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:55.851164 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerStarted","Data":"169d99f8d9d711710aec7cda8317baed6f984ede7ef7d0a6808e27ac511e1f4f"} Oct 30 21:28:55.851304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:55.851288 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:28:55.974123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:55.974070 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" podStartSLOduration=1.776792687 podStartE2EDuration="4.974055039s" totalImagesPullingTime="3.197262352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:55.972845653 +0000 UTC m=+360.646892190" watchObservedRunningTime="2026-10-30 21:28:55.974055039 +0000 UTC m=+360.648101574" Oct 30 21:28:56.092815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:56.092762 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-685458bf9d-bz826" podStartSLOduration=2.158815377 podStartE2EDuration="5.092748056s" totalImagesPullingTime="2.933932679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:56.091185073 +0000 UTC m=+360.765231610" watchObservedRunningTime="2026-10-30 21:28:56.092748056 +0000 UTC m=+360.766794615" Oct 30 21:28:58.687862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.687828 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76"] Oct 30 21:28:58.691278 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.691259 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.743786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.743749 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 21:28:58.794058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.794020 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 21:28:58.794058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.794041 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-plct4\"" Oct 30 21:28:58.794291 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.794059 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 21:28:58.796213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.796188 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76"] Oct 30 21:28:58.818525 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.818494 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.818692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.818537 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.818692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.818653 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zwt\" (UniqueName: \"kubernetes.io/projected/6123e7bb-86b6-417c-9961-0d62abd097b8-kube-api-access-c9zwt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.818765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.818711 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.920158 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.920119 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.920341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.920182 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.920341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.920213 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.920341 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:58.920251 4525 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 21:28:58.920341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.920266 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zwt\" (UniqueName: \"kubernetes.io/projected/6123e7bb-86b6-417c-9961-0d62abd097b8-kube-api-access-c9zwt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.920341 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:58.920322 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source podName:6123e7bb-86b6-417c-9961-0d62abd097b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.42030476 +0000 UTC m=+364.094351298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-bsp76" (UID: "6123e7bb-86b6-417c-9961-0d62abd097b8") : configmap "usage-tenancy-proxy-source" not found Oct 30 21:28:58.920654 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:58.920340 4525 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 21:28:58.920654 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:28:58.920412 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert podName:6123e7bb-86b6-417c-9961-0d62abd097b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.420397554 +0000 UTC m=+364.094444077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-bsp76" (UID: "6123e7bb-86b6-417c-9961-0d62abd097b8") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 21:28:58.920969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.920946 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:58.981663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:58.981574 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zwt\" (UniqueName: \"kubernetes.io/projected/6123e7bb-86b6-417c-9961-0d62abd097b8-kube-api-access-c9zwt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.424484 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.424435 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.424709 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.424508 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.425278 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.425256 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6123e7bb-86b6-417c-9961-0d62abd097b8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.427029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.427006 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6123e7bb-86b6-417c-9961-0d62abd097b8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-bsp76\" (UID: \"6123e7bb-86b6-417c-9961-0d62abd097b8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.603078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.603033 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:28:59.975834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:28:59.975813 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76"] Oct 30 21:28:59.978314 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:28:59.978282 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6123e7bb_86b6_417c_9961_0d62abd097b8.slice/crio-1d7a00cfcc1fd3a8f49d5c406d578202abdf6e589e610666095a73cd9d2c15cd WatchSource:0}: Error finding container 1d7a00cfcc1fd3a8f49d5c406d578202abdf6e589e610666095a73cd9d2c15cd: Status 404 returned error can't find the container with id 1d7a00cfcc1fd3a8f49d5c406d578202abdf6e589e610666095a73cd9d2c15cd Oct 30 21:29:00.872980 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:00.872943 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" event={"ID":"6123e7bb-86b6-417c-9961-0d62abd097b8","Type":"ContainerStarted","Data":"1d7a00cfcc1fd3a8f49d5c406d578202abdf6e589e610666095a73cd9d2c15cd"} Oct 30 21:29:05.398352 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:29:05.398314 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:29:06.857509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:06.857475 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:29:06.858133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:06.857568 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:29:07.027875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.027845 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:29:07.028084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.028036 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" podUID="a65264b5-3775-4cfd-8cb8-10f015d801a0" containerName="manager" containerID="cri-o://a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e" gracePeriod=30 Oct 30 21:29:07.297912 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.297882 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:29:07.402801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.402721 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\") pod \"a65264b5-3775-4cfd-8cb8-10f015d801a0\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " Oct 30 21:29:07.402801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.402783 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\" (UniqueName: \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\") pod \"a65264b5-3775-4cfd-8cb8-10f015d801a0\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " Oct 30 21:29:07.403031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.402899 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\") pod \"a65264b5-3775-4cfd-8cb8-10f015d801a0\" (UID: \"a65264b5-3775-4cfd-8cb8-10f015d801a0\") " Oct 30 21:29:07.405161 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.405127 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert" pod "a65264b5-3775-4cfd-8cb8-10f015d801a0" (UID: "a65264b5-3775-4cfd-8cb8-10f015d801a0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:07.405292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.405216 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2" pod "a65264b5-3775-4cfd-8cb8-10f015d801a0" (UID: "a65264b5-3775-4cfd-8cb8-10f015d801a0"). InnerVolumeSpecName "kube-api-access-wrzg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:07.405292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.405222 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs" pod "a65264b5-3775-4cfd-8cb8-10f015d801a0" (UID: "a65264b5-3775-4cfd-8cb8-10f015d801a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:07.504010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.503971 4525 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-webhook-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:07.504010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.503998 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65264b5-3775-4cfd-8cb8-10f015d801a0-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:07.504010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.504012 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrzg2\" (UniqueName: \"kubernetes.io/projected/a65264b5-3775-4cfd-8cb8-10f015d801a0-kube-api-access-wrzg2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:07.907739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.907704 4525 generic.go:356] "Generic (PLEG): container finished" podID="a65264b5-3775-4cfd-8cb8-10f015d801a0" containerID="a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e" exitCode=0 Oct 30 21:29:07.908243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.907773 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" Oct 30 21:29:07.908243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.907800 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" event={"ID":"a65264b5-3775-4cfd-8cb8-10f015d801a0","Type":"ContainerDied","Data":"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e"} Oct 30 21:29:07.908243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.907870 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rr2sq" event={"ID":"a65264b5-3775-4cfd-8cb8-10f015d801a0","Type":"ContainerDied","Data":"6def7dc3ae108ff118a3d5ccc583ef83d9065645dd1977a448f94c7b62bfdad7"} Oct 30 21:29:07.908243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.907895 4525 scope.go:122] "RemoveContainer" containerID="a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e" Oct 30 21:29:07.917533 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.917509 4525 scope.go:122] "RemoveContainer" containerID="a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e" Oct 30 21:29:07.917879 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:29:07.917853 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e\": container with ID starting with a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e not found: ID does not exist" containerID="a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e" Oct 30 21:29:07.917974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:07.917893 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e"} err="failed to get container status \"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e\": rpc error: code = NotFound desc = could not find container \"a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e\": container with ID starting with a7e80c5f034e7172a154806497e2ecece2493ded429b910a6e38cfcde3d4049e not found: ID does not exist" Oct 30 21:29:08.092882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:08.092848 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:29:08.111111 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:08.111078 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rr2sq"] Oct 30 21:29:09.482788 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:09.473986 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65264b5-3775-4cfd-8cb8-10f015d801a0" path="/var/lib/kubelet/pods/a65264b5-3775-4cfd-8cb8-10f015d801a0/volumes" Oct 30 21:29:13.683534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.683501 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz"] Oct 30 21:29:13.683916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.683816 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a65264b5-3775-4cfd-8cb8-10f015d801a0" containerName="manager" Oct 30 21:29:13.683916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.683827 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65264b5-3775-4cfd-8cb8-10f015d801a0" containerName="manager" Oct 30 21:29:13.683916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.683905 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65264b5-3775-4cfd-8cb8-10f015d801a0" containerName="manager" Oct 30 21:29:13.686775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.686760 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.793060 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.793036 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz"] Oct 30 21:29:13.855705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855668 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.855705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855713 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.855875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855741 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.855908 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855873 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.855946 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855905 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgg2w\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-kube-api-access-qgg2w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.856001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.855946 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.856070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.856049 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.938874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.938785 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" event={"ID":"6123e7bb-86b6-417c-9961-0d62abd097b8","Type":"ContainerStarted","Data":"bd8dfe1cead6630e5980047193027ed6bf5e3246138014e4dba0966a29999892"} Oct 30 21:29:13.939029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.938901 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:29:13.956936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.956904 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.956944 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgg2w\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-kube-api-access-qgg2w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.956978 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.957009 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.957037 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.957061 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.957339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.957099 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.958104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.958072 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.959429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.959408 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.959628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.959584 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.959920 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.959905 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:13.959987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:13.959972 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:14.018408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.018372 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:14.019420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.019400 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgg2w\" (UniqueName: \"kubernetes.io/projected/93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1-kube-api-access-qgg2w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ktljz\" (UID: \"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:14.060134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.060080 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" podStartSLOduration=2.263043718 podStartE2EDuration="16.060066919s" totalImagesPullingTime="13.797023201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:14.058653441 +0000 UTC m=+378.732699978" watchObservedRunningTime="2026-10-30 21:29:14.060066919 +0000 UTC m=+378.734113455" Oct 30 21:29:14.297736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.297651 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:14.757308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.757284 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz"] Oct 30 21:29:14.759663 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:29:14.759633 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b1522f_f6a1_4cb1_9a10_187bbc7e2ca1.slice/crio-5069cec046976c34ce67126378170ecb98f9f2ad62477e38e378c377fca2ab68 WatchSource:0}: Error finding container 5069cec046976c34ce67126378170ecb98f9f2ad62477e38e378c377fca2ab68: Status 404 returned error can't find the container with id 5069cec046976c34ce67126378170ecb98f9f2ad62477e38e378c377fca2ab68 Oct 30 21:29:14.761763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.761735 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:29:14.761830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.761804 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:29:14.945179 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.945145 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" event={"ID":"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1","Type":"ContainerStarted","Data":"8c1ceace9d854ee8ff43857f62ddb6f037269812ab7518e60644f0d761fab345"} Oct 30 21:29:14.945345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.945197 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" event={"ID":"93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1","Type":"ContainerStarted","Data":"5069cec046976c34ce67126378170ecb98f9f2ad62477e38e378c377fca2ab68"} Oct 30 21:29:14.945345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:14.945230 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:15.067489 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:15.067341 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" podStartSLOduration=2.067324678 podStartE2EDuration="2.067324678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:15.066911523 +0000 UTC m=+379.740958060" watchObservedRunningTime="2026-10-30 21:29:15.067324678 +0000 UTC m=+379.741371215" Oct 30 21:29:15.950806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:15.950775 4525 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ktljz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:29:15.951234 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:15.950824 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" podUID="93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:29:18.950542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:18.950511 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ktljz" Oct 30 21:29:19.158014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.157984 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:29:19.158261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.158237 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerName="discovery" containerID="cri-o://559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69" gracePeriod=30 Oct 30 21:29:19.407212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.407188 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:29:19.504991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.504912 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.504991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.504952 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.504991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.504993 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.505197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.505020 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.505197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.505092 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.505197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.505132 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.505197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.505169 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\") pod \"f852fe6d-f39b-46c0-8610-7faf8383ced0\" (UID: \"f852fe6d-f39b-46c0-8610-7faf8383ced0\") " Oct 30 21:29:19.505747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.505713 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:19.507652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507598 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "kube-api-access-knbrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:19.507747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507673 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:19.507812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507772 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:19.507812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507782 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:19.507944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507926 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:19.507977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.507954 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token" pod "f852fe6d-f39b-46c0-8610-7faf8383ced0" (UID: "f852fe6d-f39b-46c0-8610-7faf8383ced0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:19.606053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606018 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knbrb\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-kube-api-access-knbrb\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606045 4525 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-cacerts\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606058 4525 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-ca-configmap\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606072 4525 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-kubeconfig\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606084 4525 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f852fe6d-f39b-46c0-8610-7faf8383ced0-local-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606095 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.606292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.606105 4525 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f852fe6d-f39b-46c0-8610-7faf8383ced0-istio-csr-dns-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:19.966534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.966500 4525 generic.go:356] "Generic (PLEG): container finished" podID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerID="559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69" exitCode=0 Oct 30 21:29:19.966971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.966543 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" Oct 30 21:29:19.966971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.966577 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" event={"ID":"f852fe6d-f39b-46c0-8610-7faf8383ced0","Type":"ContainerDied","Data":"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69"} Oct 30 21:29:19.966971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.966644 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4" event={"ID":"f852fe6d-f39b-46c0-8610-7faf8383ced0","Type":"ContainerDied","Data":"ab2703b8262ac00fbaeafdbcad25352418c5f16187d623ecb1a5840fffbc74c8"} Oct 30 21:29:19.966971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.966664 4525 scope.go:122] "RemoveContainer" containerID="559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69" Oct 30 21:29:19.975560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.975543 4525 scope.go:122] "RemoveContainer" containerID="559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69" Oct 30 21:29:19.975871 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:29:19.975855 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69\": container with ID starting with 559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69 not found: ID does not exist" containerID="559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69" Oct 30 21:29:19.975921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:19.975881 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69"} err="failed to get container status \"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69\": rpc error: code = NotFound desc = could not find container \"559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69\": container with ID starting with 559119eb4f1a45a9a0fa6c7ab1da8aa61deb2c0414f243658b741b6525f14b69 not found: ID does not exist" Oct 30 21:29:20.131306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:20.131275 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:29:20.181516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:20.181485 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-qcnx4"] Oct 30 21:29:21.471966 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:21.471935 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" path="/var/lib/kubelet/pods/f852fe6d-f39b-46c0-8610-7faf8383ced0/volumes" Oct 30 21:29:24.952044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:24.952016 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-bsp76" Oct 30 21:29:28.498362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.498326 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:28.498808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.498680 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerName="discovery" Oct 30 21:29:28.498808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.498692 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerName="discovery" Oct 30 21:29:28.498808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.498766 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f852fe6d-f39b-46c0-8610-7faf8383ced0" containerName="discovery" Oct 30 21:29:28.502804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.502784 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.556098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.556057 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:29:28.582765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.582731 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.582765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.582768 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczjf\" (UniqueName: \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.582960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.582898 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.582960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.582956 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.605863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.605811 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:29:28.606255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.606238 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:29:28.606359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.606274 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k79xr\"" Oct 30 21:29:28.607813 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.607792 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:28.656072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.656041 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:29:28.683551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.683513 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.683784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.683580 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.683784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.683647 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.683784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.683675 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczjf\" (UniqueName: \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.684358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.684336 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.686323 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.686300 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.686415 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.686341 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.745536 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.745498 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczjf\" (UniqueName: \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\") pod \"maas-api-6d669c7fcd-4gzv4\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:28.814795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:28.814702 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:29.185967 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:29.185928 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:30.011626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:30.011564 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" event={"ID":"bbd73818-be7c-4dcc-82c5-2dc8f215bc88","Type":"ContainerStarted","Data":"f6f4dd558e966b3dca4247391eef5d2159f5e6269dacaf7882c1b77f797c95d8"} Oct 30 21:29:32.021472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:32.021434 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" event={"ID":"bbd73818-be7c-4dcc-82c5-2dc8f215bc88","Type":"ContainerStarted","Data":"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1"} Oct 30 21:29:32.021955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:32.021632 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:32.140123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:32.140070 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" podStartSLOduration=2.227066649 podStartE2EDuration="4.140054469s" totalImagesPullingTime="1.91298782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:32.139066969 +0000 UTC m=+396.813113505" watchObservedRunningTime="2026-10-30 21:29:32.140054469 +0000 UTC m=+396.814101005" Oct 30 21:29:35.395488 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:29:35.395446 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:29:36.252413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:36.252333 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:29:38.032056 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:38.032029 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:39.138542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.138502 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 21:29:39.260477 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.260425 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 21:29:39.260666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.260568 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.375238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.375203 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.375238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.375243 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.375450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.375275 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.375450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.375342 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjbg\" (UniqueName: \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.476331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.476253 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.476331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.476291 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.476331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.476322 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.476679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.476340 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjbg\" (UniqueName: \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.476960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.476928 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.478954 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.478932 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.479026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.479011 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.537841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.537806 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjbg\" (UniqueName: \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\") pod \"maas-api-5fd68d57f5-hqqfd\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.583659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.583602 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:39.989145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:39.989121 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 21:29:39.991702 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:29:39.991663 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014bcc65_dedd_4cdb_8c95_210fdc24ffe7.slice/crio-30f85a4b1e4951b9cd33538e7a4a2868baa23d45146a591d53e618284700e99e WatchSource:0}: Error finding container 30f85a4b1e4951b9cd33538e7a4a2868baa23d45146a591d53e618284700e99e: Status 404 returned error can't find the container with id 30f85a4b1e4951b9cd33538e7a4a2868baa23d45146a591d53e618284700e99e Oct 30 21:29:40.058208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:40.058166 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" event={"ID":"014bcc65-dedd-4cdb-8c95-210fdc24ffe7","Type":"ContainerStarted","Data":"30f85a4b1e4951b9cd33538e7a4a2868baa23d45146a591d53e618284700e99e"} Oct 30 21:29:42.067929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:42.067897 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" event={"ID":"014bcc65-dedd-4cdb-8c95-210fdc24ffe7","Type":"ContainerStarted","Data":"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466"} Oct 30 21:29:42.068344 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:42.067994 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:42.186927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:42.186873 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" podStartSLOduration=2.027775113 podStartE2EDuration="3.186856317s" totalImagesPullingTime="1.159081204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:42.1856339 +0000 UTC m=+406.859680435" watchObservedRunningTime="2026-10-30 21:29:42.186856317 +0000 UTC m=+406.860902852" Oct 30 21:29:48.077887 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.077857 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 21:29:48.247511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.247472 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:48.247755 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.247727 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" podUID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" containerName="maas-api" containerID="cri-o://cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1" gracePeriod=30 Oct 30 21:29:48.492195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.492171 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:48.556864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.556833 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\") pod \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " Oct 30 21:29:48.557036 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.556876 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\" (UniqueName: \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\") pod \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " Oct 30 21:29:48.557036 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.556919 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\") pod \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " Oct 30 21:29:48.557036 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.556947 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\") pod \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\" (UID: \"bbd73818-be7c-4dcc-82c5-2dc8f215bc88\") " Oct 30 21:29:48.557396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.557358 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca" pod "bbd73818-be7c-4dcc-82c5-2dc8f215bc88" (UID: "bbd73818-be7c-4dcc-82c5-2dc8f215bc88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:48.559398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.559376 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs" pod "bbd73818-be7c-4dcc-82c5-2dc8f215bc88" (UID: "bbd73818-be7c-4dcc-82c5-2dc8f215bc88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:48.559691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.559671 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls" pod "bbd73818-be7c-4dcc-82c5-2dc8f215bc88" (UID: "bbd73818-be7c-4dcc-82c5-2dc8f215bc88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:48.559766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.559689 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf" pod "bbd73818-be7c-4dcc-82c5-2dc8f215bc88" (UID: "bbd73818-be7c-4dcc-82c5-2dc8f215bc88"). InnerVolumeSpecName "kube-api-access-dczjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:48.658534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.658498 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczjf\" (UniqueName: \"kubernetes.io/projected/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-kube-api-access-dczjf\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:48.658534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.658528 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:48.658534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.658538 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:48.658789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:48.658546 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbd73818-be7c-4dcc-82c5-2dc8f215bc88-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:49.102576 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.102498 4525 generic.go:356] "Generic (PLEG): container finished" podID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" containerID="cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1" exitCode=0 Oct 30 21:29:49.102576 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.102552 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" Oct 30 21:29:49.103070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.102577 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" event={"ID":"bbd73818-be7c-4dcc-82c5-2dc8f215bc88","Type":"ContainerDied","Data":"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1"} Oct 30 21:29:49.103070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.102684 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4" event={"ID":"bbd73818-be7c-4dcc-82c5-2dc8f215bc88","Type":"ContainerDied","Data":"f6f4dd558e966b3dca4247391eef5d2159f5e6269dacaf7882c1b77f797c95d8"} Oct 30 21:29:49.103070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.102714 4525 scope.go:122] "RemoveContainer" containerID="cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1" Oct 30 21:29:49.111409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.111390 4525 scope.go:122] "RemoveContainer" containerID="cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1" Oct 30 21:29:49.111741 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:29:49.111724 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1\": container with ID starting with cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1 not found: ID does not exist" containerID="cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1" Oct 30 21:29:49.111806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.111752 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1"} err="failed to get container status \"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1\": rpc error: code = NotFound desc = could not find container \"cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1\": container with ID starting with cdb162a073650a44031f086bd37477452ece949f431fa596aa2eaf1d2d7fcdb1 not found: ID does not exist" Oct 30 21:29:49.268487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.268446 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:49.318662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.318631 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4gzv4"] Oct 30 21:29:49.471252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:29:49.471218 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" path="/var/lib/kubelet/pods/bbd73818-be7c-4dcc-82c5-2dc8f215bc88/volumes" Oct 30 21:30:05.395117 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:30:05.395080 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:30:35.395715 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:30:35.395677 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:31:05.395171 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:31:05.395129 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:31:23.491071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:23.491038 4525 generic.go:356] "Generic (PLEG): container finished" podID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerID="169d99f8d9d711710aec7cda8317baed6f984ede7ef7d0a6808e27ac511e1f4f" exitCode=1 Oct 30 21:31:23.491497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:23.491112 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerDied","Data":"169d99f8d9d711710aec7cda8317baed6f984ede7ef7d0a6808e27ac511e1f4f"} Oct 30 21:31:23.491497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:23.491458 4525 scope.go:122] "RemoveContainer" containerID="169d99f8d9d711710aec7cda8317baed6f984ede7ef7d0a6808e27ac511e1f4f" Oct 30 21:31:25.502444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:25.502410 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerStarted","Data":"31c0773591e788ae42c4bb4cd39e101c0c4cb4dc6d5b0c2b09081e298d907061"} Oct 30 21:31:25.502869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:25.502648 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:31:35.219305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.219269 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8"] Oct 30 21:31:35.219833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.219595 4525 cpu_manager.go:414] "containerMap: removing container" podUID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" containerName="maas-api" Oct 30 21:31:35.219833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.219619 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" containerName="maas-api" Oct 30 21:31:35.219833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.219687 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd73818-be7c-4dcc-82c5-2dc8f215bc88" containerName="maas-api" Oct 30 21:31:35.222577 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.222561 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.254823 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.254786 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxklq\" (UniqueName: \"kubernetes.io/projected/35cb03d2-e469-4853-a4c1-8ec647eba836-kube-api-access-vxklq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.254979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.254852 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35cb03d2-e469-4853-a4c1-8ec647eba836-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.255037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.255010 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.255075 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.255058 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.255200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.255174 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.255238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.255220 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.325157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.325113 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:31:35.325364 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.325294 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:31:35.325364 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.325317 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:31:35.325460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.325363 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lt9rb\"" Oct 30 21:31:35.326728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.326713 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8"] Oct 30 21:31:35.356688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356654 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35cb03d2-e469-4853-a4c1-8ec647eba836-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.356834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356714 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.356834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356745 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.356834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356787 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.356976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356933 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.357034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.356983 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxklq\" (UniqueName: \"kubernetes.io/projected/35cb03d2-e469-4853-a4c1-8ec647eba836-kube-api-access-vxklq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.357289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.357256 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.357289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.357286 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.357477 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.357413 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.359255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.359236 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35cb03d2-e469-4853-a4c1-8ec647eba836-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.359384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.359368 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35cb03d2-e469-4853-a4c1-8ec647eba836-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.397066 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:31:35.397029 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:31:35.420779 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.420738 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxklq\" (UniqueName: \"kubernetes.io/projected/35cb03d2-e469-4853-a4c1-8ec647eba836-kube-api-access-vxklq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8\" (UID: \"35cb03d2-e469-4853-a4c1-8ec647eba836\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.535046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.534960 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:35.908824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:35.908654 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8"] Oct 30 21:31:35.911620 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:31:35.911575 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb03d2_e469_4853_a4c1_8ec647eba836.slice/crio-b3e10559ae8314855d2b2605159064342075abd1ec0bb94ecd2f17a89423d77a WatchSource:0}: Error finding container b3e10559ae8314855d2b2605159064342075abd1ec0bb94ecd2f17a89423d77a: Status 404 returned error can't find the container with id b3e10559ae8314855d2b2605159064342075abd1ec0bb94ecd2f17a89423d77a Oct 30 21:31:36.509050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:36.509019 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 21:31:36.546464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:36.546430 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" event={"ID":"35cb03d2-e469-4853-a4c1-8ec647eba836","Type":"ContainerStarted","Data":"b3e10559ae8314855d2b2605159064342075abd1ec0bb94ecd2f17a89423d77a"} Oct 30 21:31:37.551965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:37.551930 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" event={"ID":"35cb03d2-e469-4853-a4c1-8ec647eba836","Type":"ContainerStarted","Data":"4858c794ea26720b62980c29b80f357f17aa79bd3a445013616a2571d016ad63"} Oct 30 21:31:37.671270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:37.671213 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" podStartSLOduration=1.116966748 podStartE2EDuration="2.671195004s" totalImagesPullingTime="1.554228256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:37.669794457 +0000 UTC m=+522.343840994" watchObservedRunningTime="2026-10-30 21:31:37.671195004 +0000 UTC m=+522.345241539" Oct 30 21:31:38.536125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:38.536090 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:38.749404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:38.748894 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:31:39.365740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.365705 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn"] Oct 30 21:31:39.369126 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.369107 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.471231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.471201 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:31:39.472881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.472858 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn"] Oct 30 21:31:39.494150 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494109 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcn8q\" (UniqueName: \"kubernetes.io/projected/71e95dbd-f2d9-47d1-812d-111c027f4ef8-kube-api-access-vcn8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.494314 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494168 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.494314 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494188 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.494396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494314 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.494396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494346 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.494396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.494364 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595450 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcn8q\" (UniqueName: \"kubernetes.io/projected/71e95dbd-f2d9-47d1-812d-111c027f4ef8-kube-api-access-vcn8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595551 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595584 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595660 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595699 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.595798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.595722 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.596083 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.596064 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.596143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.596121 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.596186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.596166 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.598134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.598105 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e95dbd-f2d9-47d1-812d-111c027f4ef8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.598420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.598401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e95dbd-f2d9-47d1-812d-111c027f4ef8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.669978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.669935 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcn8q\" (UniqueName: \"kubernetes.io/projected/71e95dbd-f2d9-47d1-812d-111c027f4ef8-kube-api-access-vcn8q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn\" (UID: \"71e95dbd-f2d9-47d1-812d-111c027f4ef8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:39.681812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:39.681775 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:40.054465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:40.054237 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn"] Oct 30 21:31:40.057011 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:31:40.056979 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e95dbd_f2d9_47d1_812d_111c027f4ef8.slice/crio-966474ac533e2ef64c87aedfe89707a22f862c2f1cedf2c70105c57b03bf498c WatchSource:0}: Error finding container 966474ac533e2ef64c87aedfe89707a22f862c2f1cedf2c70105c57b03bf498c: Status 404 returned error can't find the container with id 966474ac533e2ef64c87aedfe89707a22f862c2f1cedf2c70105c57b03bf498c Oct 30 21:31:40.565272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:40.565236 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" event={"ID":"71e95dbd-f2d9-47d1-812d-111c027f4ef8","Type":"ContainerStarted","Data":"966474ac533e2ef64c87aedfe89707a22f862c2f1cedf2c70105c57b03bf498c"} Oct 30 21:31:41.570459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:41.570428 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" event={"ID":"71e95dbd-f2d9-47d1-812d-111c027f4ef8","Type":"ContainerStarted","Data":"a488eaf6520a8615ce3c41752126a8820a3ed44b9af9e476c5f7482404a86a8d"} Oct 30 21:31:41.682840 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:41.682806 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:41.689993 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:41.689949 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" podStartSLOduration=2.094603982 podStartE2EDuration="2.689932027s" totalImagesPullingTime="595.328045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:41.68913064 +0000 UTC m=+526.363177190" watchObservedRunningTime="2026-10-30 21:31:41.689932027 +0000 UTC m=+526.363978566" Oct 30 21:31:45.535955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:45.535915 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:45.548885 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:45.548855 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:45.602726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:45.602697 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8" Oct 30 21:31:47.222334 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:47.222297 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:31:49.682143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:49.682103 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:49.694713 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:49.694686 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:50.624815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:50.624787 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn" Oct 30 21:31:59.014113 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.014072 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c"] Oct 30 21:31:59.020137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.020111 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.077682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.077638 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c"] Oct 30 21:31:59.125121 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.125086 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:31:59.170050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170012 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.170229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170064 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.170229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170090 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.170229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170141 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.170347 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170315 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mgb\" (UniqueName: \"kubernetes.io/projected/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-kube-api-access-88mgb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.170381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.170353 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.270926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.270839 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mgb\" (UniqueName: \"kubernetes.io/projected/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-kube-api-access-88mgb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.270926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.270915 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.270964 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271010 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271067 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271090 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271468 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271438 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271473 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.271572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.271523 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.273559 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.273535 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.273731 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.273711 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.332298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.332261 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mgb\" (UniqueName: \"kubernetes.io/projected/5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa-kube-api-access-88mgb\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d4t8c\" (UID: \"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.332479 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.332421 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:31:59.707584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:31:59.707512 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c"] Oct 30 21:31:59.710284 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:31:59.710260 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4aff0e_b9cd_4ae2_a3bf_59dbbd6718aa.slice/crio-846b43850d4bac5fa8a92af58361b408545472af97a65a6e1567e0dd71786428 WatchSource:0}: Error finding container 846b43850d4bac5fa8a92af58361b408545472af97a65a6e1567e0dd71786428: Status 404 returned error can't find the container with id 846b43850d4bac5fa8a92af58361b408545472af97a65a6e1567e0dd71786428 Oct 30 21:32:00.654386 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:00.654354 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" event={"ID":"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa","Type":"ContainerStarted","Data":"fdd8caaa52022ca852c8c7e0677446dd6bb818731fdaa409d7897b61c7c1a05a"} Oct 30 21:32:00.654789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:00.654404 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" event={"ID":"5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa","Type":"ContainerStarted","Data":"846b43850d4bac5fa8a92af58361b408545472af97a65a6e1567e0dd71786428"} Oct 30 21:32:00.823537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:00.823479 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" podStartSLOduration=2.237499509 podStartE2EDuration="2.823462062s" totalImagesPullingTime="585.962553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:00.822327396 +0000 UTC m=+545.496373936" watchObservedRunningTime="2026-10-30 21:32:00.823462062 +0000 UTC m=+545.497508633" Oct 30 21:32:01.333259 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:01.333220 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:32:01.834840 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:01.834812 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:32:05.395443 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:32:05.395408 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:32:07.042059 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.042024 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9vnhn"] Oct 30 21:32:07.045517 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.045499 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.098088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.098059 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:32:07.146631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146586 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.146792 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146638 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.146792 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146678 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.146792 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146713 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvwc\" (UniqueName: \"kubernetes.io/projected/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-kube-api-access-jvvwc\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.146897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146788 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-home\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.146897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.146818 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.149945 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.149925 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9vnhn"] Oct 30 21:32:07.247509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247472 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.247509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247513 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.247818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247534 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.247818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247550 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvwc\" (UniqueName: \"kubernetes.io/projected/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-kube-api-access-jvvwc\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.247818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247589 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-home\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.247818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247645 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.248026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.247988 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.248093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.248038 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.248093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.248062 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-home\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.250374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.250345 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.250556 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.250530 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.309412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.309340 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvwc\" (UniqueName: \"kubernetes.io/projected/c30d62a7-8f12-41dd-aff1-2bb4d20498e3-kube-api-access-jvvwc\") pod \"echo-simulator-kserve-64b6f8f96c-9vnhn\" (UID: \"c30d62a7-8f12-41dd-aff1-2bb4d20498e3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.357161 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.357128 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:07.731007 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:07.730931 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9vnhn"] Oct 30 21:32:07.733136 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:07.733107 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30d62a7_8f12_41dd_aff1_2bb4d20498e3.slice/crio-99224526f9a06217d05dce684e0abbb091a0a428632ab46f71913247d5134d45 WatchSource:0}: Error finding container 99224526f9a06217d05dce684e0abbb091a0a428632ab46f71913247d5134d45: Status 404 returned error can't find the container with id 99224526f9a06217d05dce684e0abbb091a0a428632ab46f71913247d5134d45 Oct 30 21:32:08.689114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:08.689083 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" event={"ID":"c30d62a7-8f12-41dd-aff1-2bb4d20498e3","Type":"ContainerStarted","Data":"99224526f9a06217d05dce684e0abbb091a0a428632ab46f71913247d5134d45"} Oct 30 21:32:09.333553 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:09.333521 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:32:09.353393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:09.353353 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:32:09.705536 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:09.705509 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" Oct 30 21:32:10.700180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.700135 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" event={"ID":"c30d62a7-8f12-41dd-aff1-2bb4d20498e3","Type":"ContainerStarted","Data":"845860620c6b74909ddc43f0f11d5a48355644bddc31a68a70b33f4004967e60"} Oct 30 21:32:10.804357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.804321 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:32:10.807651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.807633 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.833861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.833808 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" podStartSLOduration=2.304831894 podStartE2EDuration="4.833792842s" totalImagesPullingTime="2.528960948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:10.817812534 +0000 UTC m=+555.491859071" watchObservedRunningTime="2026-10-30 21:32:10.833792842 +0000 UTC m=+555.507839378" Oct 30 21:32:10.862503 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.862472 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lfxgk\"" Oct 30 21:32:10.884022 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.883992 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884030 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884054 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884112 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884141 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68cs\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884267 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884301 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884330 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.884393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.884349 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.892210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.892181 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:32:10.985019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.984937 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.984971 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985106 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985160 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985193 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985220 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985246 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68cs\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985307 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985320 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985379 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985462 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985531 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985679 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.985911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.985891 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.988119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.988093 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:10.988249 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:10.988230 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:11.047357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.047316 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:11.047497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.047383 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68cs\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\") pod \"maas-default-gateway-openshift-default-75d7c785f-52mqg\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:11.120521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.120479 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:11.357858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.357776 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:11.547850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.547737 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:32:11.550114 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:11.550089 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b14672_aa4e_4179_a9c4_3c01024fcb3a.slice/crio-0cb1194aac6f2d69ddff8bdde8472d3af5fd6534cb38fefbca2076b1634b52dc WatchSource:0}: Error finding container 0cb1194aac6f2d69ddff8bdde8472d3af5fd6534cb38fefbca2076b1634b52dc: Status 404 returned error can't find the container with id 0cb1194aac6f2d69ddff8bdde8472d3af5fd6534cb38fefbca2076b1634b52dc Oct 30 21:32:11.706599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:11.706564 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" event={"ID":"72b14672-aa4e-4179-a9c4-3c01024fcb3a","Type":"ContainerStarted","Data":"0cb1194aac6f2d69ddff8bdde8472d3af5fd6534cb38fefbca2076b1634b52dc"} Oct 30 21:32:15.865444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:15.865281 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:15.865444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:15.865346 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:15.865444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:15.865378 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:16.750632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:16.748535 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" event={"ID":"72b14672-aa4e-4179-a9c4-3c01024fcb3a","Type":"ContainerStarted","Data":"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b"} Oct 30 21:32:16.923928 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:16.923872 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podStartSLOduration=2.61188881 podStartE2EDuration="6.923854649s" totalImagesPullingTime="4.311965839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:16.923010544 +0000 UTC m=+561.597057081" watchObservedRunningTime="2026-10-30 21:32:16.923854649 +0000 UTC m=+561.597901186" Oct 30 21:32:17.120680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.120647 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:17.122266 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.122240 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:17.122372 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.122292 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:17.358179 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.358146 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:17.371267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.371183 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:17.770169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:17.770099 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9vnhn" Oct 30 21:32:18.121554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:18.121527 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:18.121960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:18.121582 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:19.120896 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:19.120852 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:19.121081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:19.120949 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:19.551694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:19.551456 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:32:20.121210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:20.121169 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:20.121391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:20.121240 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:21.121329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:21.121289 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:21.121764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:21.121348 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:22.121742 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:22.121699 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:22.122143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:22.121770 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:23.121457 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:23.121407 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-52mqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:32:23.121457 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:23.121459 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 21:32:24.137555 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:24.137525 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:24.138050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:24.137975 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:24.138833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:24.138815 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:32:31.838396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.838361 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:32:31.842024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.842006 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:31.894400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.894353 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:31.894582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.894418 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk48k\" (UniqueName: \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:31.894694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.894679 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:32:31.894751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.894683 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5wltk\"" Oct 30 21:32:31.895014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.894996 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:32:31.901072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.901031 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:32:31.995038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.995008 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:31.995038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.995047 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk48k\" (UniqueName: \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:31.997807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:31.997780 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:32.056339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:32.056304 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk48k\" (UniqueName: \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\") pod \"ai-gateway-controller-6d49f5bf5d-9r24h\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:32.154984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:32.154949 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:32.525749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:32.525725 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:32:32.528055 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:32.528030 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6118bd20_ec3b_408a_a3c3_9a60cfdfe3cc.slice/crio-f2e79138edeb189530828df70a1c77d1e188d663025290562419a258efadae59 WatchSource:0}: Error finding container f2e79138edeb189530828df70a1c77d1e188d663025290562419a258efadae59: Status 404 returned error can't find the container with id f2e79138edeb189530828df70a1c77d1e188d663025290562419a258efadae59 Oct 30 21:32:32.826988 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:32.826899 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" event={"ID":"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc","Type":"ContainerStarted","Data":"f2e79138edeb189530828df70a1c77d1e188d663025290562419a258efadae59"} Oct 30 21:32:35.396507 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:32:35.396466 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:32:39.858996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:39.858956 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" event={"ID":"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc","Type":"ContainerStarted","Data":"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427"} Oct 30 21:32:39.859428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:39.859056 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:39.976203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:39.976149 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" podStartSLOduration=2.665864683 podStartE2EDuration="8.976134358s" totalImagesPullingTime="6.310269675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:39.975334214 +0000 UTC m=+584.649380751" watchObservedRunningTime="2026-10-30 21:32:39.976134358 +0000 UTC m=+584.650180894" Oct 30 21:32:42.075316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.075285 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g"] Oct 30 21:32:42.081733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.081712 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.136046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.135525 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:32:42.136046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.135529 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g8zp7\"" Oct 30 21:32:42.137666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.137639 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g"] Oct 30 21:32:42.191267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.191231 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed476948-8416-4669-9229-6cf4011aa55c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.191267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.191271 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffkl\" (UniqueName: \"kubernetes.io/projected/ed476948-8416-4669-9229-6cf4011aa55c-kube-api-access-lffkl\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.292591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.292556 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffkl\" (UniqueName: \"kubernetes.io/projected/ed476948-8416-4669-9229-6cf4011aa55c-kube-api-access-lffkl\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.292800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.292699 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed476948-8416-4669-9229-6cf4011aa55c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.293373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.293353 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed476948-8416-4669-9229-6cf4011aa55c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.354397 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.354306 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffkl\" (UniqueName: \"kubernetes.io/projected/ed476948-8416-4669-9229-6cf4011aa55c-kube-api-access-lffkl\") pod \"payload-pre-processing-7b98bb678c-6qs7g\" (UID: \"ed476948-8416-4669-9229-6cf4011aa55c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.394156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.394110 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:42.766535 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.766502 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g"] Oct 30 21:32:42.768767 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:42.768735 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded476948_8416_4669_9229_6cf4011aa55c.slice/crio-afc98bf65351990acc75061822ad6ec8e525c947c7223a025784a3f55fec1c8d WatchSource:0}: Error finding container afc98bf65351990acc75061822ad6ec8e525c947c7223a025784a3f55fec1c8d: Status 404 returned error can't find the container with id afc98bf65351990acc75061822ad6ec8e525c947c7223a025784a3f55fec1c8d Oct 30 21:32:42.872911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:42.872871 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" event={"ID":"ed476948-8416-4669-9229-6cf4011aa55c","Type":"ContainerStarted","Data":"afc98bf65351990acc75061822ad6ec8e525c947c7223a025784a3f55fec1c8d"} Oct 30 21:32:45.893320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:45.893284 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" event={"ID":"ed476948-8416-4669-9229-6cf4011aa55c","Type":"ContainerStarted","Data":"39efd5cfaddf6fb8fa9b9cbff8b3c4563687d208e668da1c5a287cee730014a4"} Oct 30 21:32:45.893752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:45.893429 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:46.009836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:46.009783 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" podStartSLOduration=1.280157057 podStartE2EDuration="4.009767483s" totalImagesPullingTime="2.729610426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:46.008963824 +0000 UTC m=+590.683010361" watchObservedRunningTime="2026-10-30 21:32:46.009767483 +0000 UTC m=+590.683814030" Oct 30 21:32:50.865225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:50.865199 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:32:51.236986 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.236949 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-8ff4cff99-w25dw"] Oct 30 21:32:51.244703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.244679 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.299385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.299351 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8ff4cff99-w25dw"] Oct 30 21:32:51.374425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.374397 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-metrics-certs\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.374630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.374454 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhk6h\" (UniqueName: \"kubernetes.io/projected/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-kube-api-access-rhk6h\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.475114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.475078 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhk6h\" (UniqueName: \"kubernetes.io/projected/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-kube-api-access-rhk6h\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.475292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.475191 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-metrics-certs\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.477893 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.477866 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-metrics-certs\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.536662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.536567 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhk6h\" (UniqueName: \"kubernetes.io/projected/a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff-kube-api-access-rhk6h\") pod \"ai-gateway-controller-8ff4cff99-w25dw\" (UID: \"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff\") " pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.557467 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.557437 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:51.929732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:51.929709 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-8ff4cff99-w25dw"] Oct 30 21:32:51.932572 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:51.932545 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84ef14f_a5a5_4b22_b1f5_8bd2c73847ff.slice/crio-a305c5c6df7a375035a2810e8039bc06a44872be0653dd0ab7d93d84eedd8eb7 WatchSource:0}: Error finding container a305c5c6df7a375035a2810e8039bc06a44872be0653dd0ab7d93d84eedd8eb7: Status 404 returned error can't find the container with id a305c5c6df7a375035a2810e8039bc06a44872be0653dd0ab7d93d84eedd8eb7 Oct 30 21:32:52.930384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:52.930351 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" event={"ID":"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff","Type":"ContainerStarted","Data":"a1ac1b57548fce02aa23a53f4d9ecc8e51385807539f761c93f74f4c34ad2401"} Oct 30 21:32:52.930797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:52.930400 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" event={"ID":"a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff","Type":"ContainerStarted","Data":"a305c5c6df7a375035a2810e8039bc06a44872be0653dd0ab7d93d84eedd8eb7"} Oct 30 21:32:52.930797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:52.930431 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:32:53.048568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:53.048464 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" podStartSLOduration=1.221594419 podStartE2EDuration="2.048438517s" totalImagesPullingTime="826.844098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:53.046692107 +0000 UTC m=+597.720738648" watchObservedRunningTime="2026-10-30 21:32:53.048438517 +0000 UTC m=+597.722485052" Oct 30 21:32:56.151227 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.151196 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:32:56.154710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.154693 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.257018 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.256990 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:32:56.317812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.317776 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.317812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.317816 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.317838 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.317908 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2cc\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.317946 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.318019 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.318048 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.318091 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.318235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.318105 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419215 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419273 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419310 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419339 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2cc\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419367 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419413 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419445 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419531 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419560 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419723 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419776 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.419947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.419845 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.420077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.420051 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.420300 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.420284 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.421953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.421924 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.422111 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.422093 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.482020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.481985 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2cc\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.482165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.481994 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-cl8lx\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.767054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.766974 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:56.901138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:56.901109 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6qs7g" Oct 30 21:32:57.195782 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.195760 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:32:57.197952 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:32:57.197925 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81086fe3_5ba5_4bec_9b22_fd4718d95727.slice/crio-9aae206c1c1c3cb86346341b19bc58ba1391fae0e2bc4572e3ba98455b92150c WatchSource:0}: Error finding container 9aae206c1c1c3cb86346341b19bc58ba1391fae0e2bc4572e3ba98455b92150c: Status 404 returned error can't find the container with id 9aae206c1c1c3cb86346341b19bc58ba1391fae0e2bc4572e3ba98455b92150c Oct 30 21:32:57.199996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.199964 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:57.200065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.200026 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:57.200065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.200054 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:32:57.960549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.960518 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" event={"ID":"81086fe3-5ba5-4bec-9b22-fd4718d95727","Type":"ContainerStarted","Data":"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1"} Oct 30 21:32:57.960855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:57.960571 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" event={"ID":"81086fe3-5ba5-4bec-9b22-fd4718d95727","Type":"ContainerStarted","Data":"9aae206c1c1c3cb86346341b19bc58ba1391fae0e2bc4572e3ba98455b92150c"} Oct 30 21:32:58.082214 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:58.082158 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podStartSLOduration=2.082142059 podStartE2EDuration="2.082142059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:58.080180507 +0000 UTC m=+602.754227043" watchObservedRunningTime="2026-10-30 21:32:58.082142059 +0000 UTC m=+602.756188594" Oct 30 21:32:58.767424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:58.767383 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:32:58.768838 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:58.768814 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:32:58.768910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:58.768870 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:32:59.767336 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:59.767299 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:32:59.767523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:32:59.767364 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:33:00.768288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:00.768247 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:33:00.768831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:00.768303 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:33:01.767712 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:01.767665 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:33:01.767899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:01.767729 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:33:02.768254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:02.768210 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:33:02.768751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:02.768278 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:33:03.767844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:03.767812 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:33:03.768026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:03.767869 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:33:03.938279 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:03.938250 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-8ff4cff99-w25dw" Oct 30 21:33:04.109231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.109152 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:33:04.109418 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.109393 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" podUID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" containerName="manager" containerID="cri-o://0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427" gracePeriod=30 Oct 30 21:33:04.357391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.357363 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:33:04.400428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.400398 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\" (UniqueName: \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\") pod \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " Oct 30 21:33:04.400657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.400488 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\") pod \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\" (UID: \"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc\") " Oct 30 21:33:04.402728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.402694 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k" pod "6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" (UID: "6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc"). InnerVolumeSpecName "kube-api-access-qk48k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:04.402831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.402702 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs" pod "6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" (UID: "6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:04.501626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.501584 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk48k\" (UniqueName: \"kubernetes.io/projected/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-kube-api-access-qk48k\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:04.501626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.501626 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:04.993799 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.993758 4525 generic.go:356] "Generic (PLEG): container finished" podID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" containerID="0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427" exitCode=0 Oct 30 21:33:04.994252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.993811 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" Oct 30 21:33:04.994252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.993853 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" event={"ID":"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc","Type":"ContainerDied","Data":"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427"} Oct 30 21:33:04.994252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.993924 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h" event={"ID":"6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc","Type":"ContainerDied","Data":"f2e79138edeb189530828df70a1c77d1e188d663025290562419a258efadae59"} Oct 30 21:33:04.994252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:04.993958 4525 scope.go:122] "RemoveContainer" containerID="0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427" Oct 30 21:33:05.005042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.005011 4525 scope.go:122] "RemoveContainer" containerID="0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427" Oct 30 21:33:05.005374 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:33:05.005349 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427\": container with ID starting with 0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427 not found: ID does not exist" containerID="0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427" Oct 30 21:33:05.005460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.005387 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427"} err="failed to get container status \"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427\": rpc error: code = NotFound desc = could not find container \"0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427\": container with ID starting with 0659faf932b3679067b9bc25ea507559905daf067e0a4346ddb54a7638b78427 not found: ID does not exist" Oct 30 21:33:05.161398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.161359 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:33:05.210483 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.210441 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6d49f5bf5d-9r24h"] Oct 30 21:33:05.397767 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:33:05.397725 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:33:05.480280 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.480237 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" path="/var/lib/kubelet/pods/6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc/volumes" Oct 30 21:33:05.530443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.530408 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:33:05.530859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.530831 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:33:05.531620 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:33:05.531586 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:33:35.396479 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:33:35.396429 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:34:05.396129 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:05.396086 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:34:16.817554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.817469 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 21:34:16.817964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.817891 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" containerName="manager" Oct 30 21:34:16.817964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.817904 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" containerName="manager" Oct 30 21:34:16.818034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.817991 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6118bd20-ec3b-408a-a3c3-9a60cfdfe3cc" containerName="manager" Oct 30 21:34:16.821033 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.821013 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.841784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841757 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.841900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841792 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.841900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841815 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.841900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841862 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.842003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841945 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbzh\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.842003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841983 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.842003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.841997 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.842108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.842055 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.842148 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.842128 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.873381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.873347 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ea374-openshift-default-dockercfg-jjxj8\"" Oct 30 21:34:16.925337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.925303 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 21:34:16.942837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.942808 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943022 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.942847 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943022 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.942863 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943022 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943008 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943053 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943091 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbzh\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943158 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943182 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943216 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943291 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943313 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943448 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943537 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.943835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.943815 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.945674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.945652 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:16.945773 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:16.945761 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:17.005053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.005019 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:17.005186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.005079 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbzh\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:17.133734 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.133699 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:17.577918 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.577874 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 21:34:17.578180 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:17.578148 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5e7f17_3426_4d62_af1a_181e70840c70.slice/crio-e69e0efe194d4e4a9f20f57ec14411620b59cb682d73e8806f5d2c58720646bb WatchSource:0}: Error finding container e69e0efe194d4e4a9f20f57ec14411620b59cb682d73e8806f5d2c58720646bb: Status 404 returned error can't find the container with id e69e0efe194d4e4a9f20f57ec14411620b59cb682d73e8806f5d2c58720646bb Oct 30 21:34:17.580642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.580598 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:17.580721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.580675 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:17.580721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.580706 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:17.830473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.830377 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:34:17.834058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.834034 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.855661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.854848 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.855901 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856194 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856248 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856282 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856452 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856538 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856570 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.856810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.856755 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9g7\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.886973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.886946 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c7d477-openshift-default-dockercfg-q6mxs\"" Oct 30 21:34:17.898897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.898863 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:34:17.958082 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958044 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958085 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958110 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958147 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958256 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958276 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958318 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9g7\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958397 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958424 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958768 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.958965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958939 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.959094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.958977 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.959094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.959034 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.959171 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.959093 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.961048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.961023 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:17.961205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:17.961189 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:18.023381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.023347 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:18.023550 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.023458 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9g7\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:18.147165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.147133 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:18.158392 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.158362 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 21:34:18.162136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.162100 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.216470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.215180 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-691598-openshift-default-dockercfg-trswq\"" Oct 30 21:34:18.232711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.231642 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 21:34:18.261852 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.261819 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.261874 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262016 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262070 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262158 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262189 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgf8\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262238 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262263 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.262390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.262366 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.327634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.327567 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerStarted","Data":"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f"} Oct 30 21:34:18.327812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.327650 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerStarted","Data":"e69e0efe194d4e4a9f20f57ec14411620b59cb682d73e8806f5d2c58720646bb"} Oct 30 21:34:18.363457 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363423 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363457 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363460 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363508 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363542 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363579 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363664 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363708 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363767 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363816 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgf8\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363921 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.363978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.363946 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.364194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.364138 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.364373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.364348 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.364457 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.364356 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.366254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.366237 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.366408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.366382 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.425225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.425147 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.425225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.425213 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgf8\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.449359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.449305 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podStartSLOduration=2.4492911299999998 podStartE2EDuration="2.44929113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:18.447251813 +0000 UTC m=+683.121298350" watchObservedRunningTime="2026-10-30 21:34:18.44929113 +0000 UTC m=+683.123337726" Oct 30 21:34:18.475315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.475269 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:18.546867 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.546840 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:34:18.548171 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:18.548146 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419137d2_3574_453c_b867_90064a172154.slice/crio-29085e9f19ab4a67e1592a37615971cf5e560ed1b9d7b2dfa0dbe87b7524ae9b WatchSource:0}: Error finding container 29085e9f19ab4a67e1592a37615971cf5e560ed1b9d7b2dfa0dbe87b7524ae9b: Status 404 returned error can't find the container with id 29085e9f19ab4a67e1592a37615971cf5e560ed1b9d7b2dfa0dbe87b7524ae9b Oct 30 21:34:18.550664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.550633 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:18.550761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.550715 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:18.550823 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.550765 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:18.853763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.853734 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 21:34:18.855312 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:18.855278 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8869ab31_ec35_487b_912f_76c5a07430ac.slice/crio-82c7a6e0bc0f5425475e9b7352b430e2fb3510e5b0cf8ed64bef82dfd2b0a9fc WatchSource:0}: Error finding container 82c7a6e0bc0f5425475e9b7352b430e2fb3510e5b0cf8ed64bef82dfd2b0a9fc: Status 404 returned error can't find the container with id 82c7a6e0bc0f5425475e9b7352b430e2fb3510e5b0cf8ed64bef82dfd2b0a9fc Oct 30 21:34:18.857830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.857800 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:18.857911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.857863 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:18.857911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:18.857895 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:34:19.136165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.136120 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:19.146509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.146474 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:19.334627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.334573 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerStarted","Data":"2d6b9e999940bfb14a2756f4dc56ba0c731af187bd701b4f4769c71627577614"} Oct 30 21:34:19.334791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.334661 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerStarted","Data":"82c7a6e0bc0f5425475e9b7352b430e2fb3510e5b0cf8ed64bef82dfd2b0a9fc"} Oct 30 21:34:19.336283 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.336256 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerStarted","Data":"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a"} Oct 30 21:34:19.336408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.336302 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerStarted","Data":"29085e9f19ab4a67e1592a37615971cf5e560ed1b9d7b2dfa0dbe87b7524ae9b"} Oct 30 21:34:19.336563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.336549 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:19.337487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.337469 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:34:19.456663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.456566 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podStartSLOduration=1.456550482 podStartE2EDuration="1.456550482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:19.455137271 +0000 UTC m=+684.129183819" watchObservedRunningTime="2026-10-30 21:34:19.456550482 +0000 UTC m=+684.130597017" Oct 30 21:34:19.475974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.475948 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:19.481249 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.481225 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:19.702787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:19.702740 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podStartSLOduration=2.702725435 podStartE2EDuration="2.702725435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:19.701483339 +0000 UTC m=+684.375529898" watchObservedRunningTime="2026-10-30 21:34:19.702725435 +0000 UTC m=+684.376771971" Oct 30 21:34:20.147452 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.147412 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:20.152949 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.152920 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:20.340976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.340945 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:20.341153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.341001 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:20.341668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.341644 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:34:20.341814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:20.341801 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:34:25.339201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.339167 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 21:34:25.342913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.342893 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.395201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.395165 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1de75a-dockercfg-n5bpx\"" Oct 30 21:34:25.441251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.441215 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfhd\" (UniqueName: \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.441425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.441281 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.445242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.445213 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1de75a\"" Oct 30 21:34:25.447147 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.447122 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 21:34:25.542256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.542222 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.542456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.542352 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfhd\" (UniqueName: \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.543044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.543022 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.604383 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.604314 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfhd\" (UniqueName: \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\") pod \"payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:25.655578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:25.655538 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:26.041587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:26.041560 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 21:34:26.043256 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:26.043232 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abb797a_663c_4e44_9903_f7797b2f6b55.slice/crio-736b02782915abfa35481cbf828fccbfb344907aed801ebfe90ebca5b44f3855 WatchSource:0}: Error finding container 736b02782915abfa35481cbf828fccbfb344907aed801ebfe90ebca5b44f3855: Status 404 returned error can't find the container with id 736b02782915abfa35481cbf828fccbfb344907aed801ebfe90ebca5b44f3855 Oct 30 21:34:26.371990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:26.371940 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" event={"ID":"4abb797a-663c-4e44-9903-f7797b2f6b55","Type":"ContainerStarted","Data":"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87"} Oct 30 21:34:26.372416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:26.372040 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:26.372416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:26.372063 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" event={"ID":"4abb797a-663c-4e44-9903-f7797b2f6b55","Type":"ContainerStarted","Data":"736b02782915abfa35481cbf828fccbfb344907aed801ebfe90ebca5b44f3855"} Oct 30 21:34:26.498374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:26.498315 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" podStartSLOduration=1.4983011259999999 podStartE2EDuration="1.498301126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:26.496800479 +0000 UTC m=+691.170847014" watchObservedRunningTime="2026-10-30 21:34:26.498301126 +0000 UTC m=+691.172347731" Oct 30 21:34:27.945956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:27.945922 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 21:34:27.949643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:27.949597 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.002032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.001995 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d68156-serving-cert\"" Oct 30 21:34:28.053827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.053790 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 21:34:28.071875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.071839 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.072056 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.071962 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdd5\" (UniqueName: \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.072123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.072061 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.072196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.072157 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.173668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.173629 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdd5\" (UniqueName: \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.173872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.173706 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.173872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.173745 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.173872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.173781 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.174051 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:28.173877 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-serving-cert: secret "maas-api-e2e-worker-w3-d68156-serving-cert" not found Oct 30 21:34:28.174051 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:28.173958 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls podName:d294fa8d-5d61-4df5-afd5-570eab9a0c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:34:28.673940055 +0000 UTC m=+693.347986572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls") pod "maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" (UID: "d294fa8d-5d61-4df5-afd5-570eab9a0c6f") : secret "maas-api-e2e-worker-w3-d68156-serving-cert" not found Oct 30 21:34:28.174471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.174446 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.176408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.176377 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.236478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.236396 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdd5\" (UniqueName: \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.679514 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.679474 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.682102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.682073 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:28.862866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:28.862830 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:29.240436 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:29.240404 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 21:34:29.242444 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:29.242402 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd294fa8d_5d61_4df5_afd5_570eab9a0c6f.slice/crio-2b23765ecbc7c43e1f40e6cb22cd8849704e0c599ff42bec4e7598f4f01673da WatchSource:0}: Error finding container 2b23765ecbc7c43e1f40e6cb22cd8849704e0c599ff42bec4e7598f4f01673da: Status 404 returned error can't find the container with id 2b23765ecbc7c43e1f40e6cb22cd8849704e0c599ff42bec4e7598f4f01673da Oct 30 21:34:29.243865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:29.243850 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:34:29.387655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:29.387621 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerStarted","Data":"2b23765ecbc7c43e1f40e6cb22cd8849704e0c599ff42bec4e7598f4f01673da"} Oct 30 21:34:30.393820 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:30.393781 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerStarted","Data":"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df"} Oct 30 21:34:30.394252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:30.393917 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:30.512720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:30.512671 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" podStartSLOduration=2.731550866 podStartE2EDuration="3.512652259s" totalImagesPullingTime="781.101393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:30.511022186 +0000 UTC m=+695.185068723" watchObservedRunningTime="2026-10-30 21:34:30.512652259 +0000 UTC m=+695.186698794" Oct 30 21:34:32.867478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.867442 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 21:34:32.871723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.871697 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:32.924804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.924765 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:32.924968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.924878 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9l2\" (UniqueName: \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:32.974210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.974176 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d68156\"" Oct 30 21:34:32.974210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.974195 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d68156-dockercfg-zmg6r\"" Oct 30 21:34:32.976180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:32.976156 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 21:34:33.026038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.026002 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9l2\" (UniqueName: \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:33.026224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.026115 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:33.026850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.026832 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:33.087745 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.087712 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9l2\" (UniqueName: \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\") pod \"payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:33.184571 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.184469 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:33.344736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.344691 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 21:34:33.349355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.349333 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.401972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.401941 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-1de75a-serving-cert\"" Oct 30 21:34:33.408559 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.408523 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 21:34:33.429892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.429861 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzj2\" (UniqueName: \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.430099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.429918 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.430099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.430031 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.430227 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.430140 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.531142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.531047 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.531142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.531094 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzj2\" (UniqueName: \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.531374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.531148 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.531374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.531180 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.531374 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:33.531290 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-serving-cert: secret "maas-api-e2e-worker-w4-1de75a-serving-cert" not found Oct 30 21:34:33.531519 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:33.531376 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls podName:30798ac3-f1bc-40bc-bc83-5914da439947 nodeName:}" failed. No retries permitted until 2026-10-30 21:34:34.031344941 +0000 UTC m=+698.705391457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls") pod "maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" (UID: "30798ac3-f1bc-40bc-bc83-5914da439947") : secret "maas-api-e2e-worker-w4-1de75a-serving-cert" not found Oct 30 21:34:33.531873 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.531855 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.533856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.533839 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:33.582060 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.582028 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 21:34:33.585215 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:33.585186 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230f0a9e_a519_4e89_8567_72086ee4c551.slice/crio-f87abfe2af0d9af89addfb4b6dc9655e2417048aff27871a2e6d133e17590ccd WatchSource:0}: Error finding container f87abfe2af0d9af89addfb4b6dc9655e2417048aff27871a2e6d133e17590ccd: Status 404 returned error can't find the container with id f87abfe2af0d9af89addfb4b6dc9655e2417048aff27871a2e6d133e17590ccd Oct 30 21:34:33.607324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:33.607295 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzj2\" (UniqueName: \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:34.037744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.037713 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:34.040409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.040384 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:34.263128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.263090 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:34.417650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.417600 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" event={"ID":"230f0a9e-a519-4e89-8567-72086ee4c551","Type":"ContainerStarted","Data":"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e"} Oct 30 21:34:34.417860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.417676 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" event={"ID":"230f0a9e-a519-4e89-8567-72086ee4c551","Type":"ContainerStarted","Data":"f87abfe2af0d9af89addfb4b6dc9655e2417048aff27871a2e6d133e17590ccd"} Oct 30 21:34:34.417860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.417807 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:34.536085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.536035 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" podStartSLOduration=2.5360201570000003 podStartE2EDuration="2.536020157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:34.534009716 +0000 UTC m=+699.208056263" watchObservedRunningTime="2026-10-30 21:34:34.536020157 +0000 UTC m=+699.210066712" Oct 30 21:34:34.696857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:34.696834 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 21:34:34.699133 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:34.699103 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30798ac3_f1bc_40bc_bc83_5914da439947.slice/crio-05d927bd4afad3b16cd01a557ef0d2d61d7453836850a7668ea4b3900930f0ed WatchSource:0}: Error finding container 05d927bd4afad3b16cd01a557ef0d2d61d7453836850a7668ea4b3900930f0ed: Status 404 returned error can't find the container with id 05d927bd4afad3b16cd01a557ef0d2d61d7453836850a7668ea4b3900930f0ed Oct 30 21:34:35.395467 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:35.395430 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:34:35.423531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:35.423477 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" event={"ID":"30798ac3-f1bc-40bc-bc83-5914da439947","Type":"ContainerStarted","Data":"05d927bd4afad3b16cd01a557ef0d2d61d7453836850a7668ea4b3900930f0ed"} Oct 30 21:34:36.400125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:36.400087 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused" Oct 30 21:34:36.431231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:36.431199 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" event={"ID":"30798ac3-f1bc-40bc-bc83-5914da439947","Type":"ContainerStarted","Data":"243851cc76e2f3edba64c15bb5b0dd4299de38f61706c68f827617d906537321"} Oct 30 21:34:36.431417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:36.431277 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:36.580463 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:36.580396 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" podStartSLOduration=2.733025454 podStartE2EDuration="3.580378849s" totalImagesPullingTime="847.353395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:36.579436162 +0000 UTC m=+701.253482697" watchObservedRunningTime="2026-10-30 21:34:36.580378849 +0000 UTC m=+701.254425386" Oct 30 21:34:37.379456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:37.379430 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 21:34:38.605635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.605590 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 21:34:38.609318 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.609293 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.661829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.661789 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-c7d477-serving-cert\"" Oct 30 21:34:38.691659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.691590 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.691827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.691720 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.691827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.691784 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2xg\" (UniqueName: \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.691947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.691920 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.713528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.713494 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 21:34:38.792482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.792448 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.792482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.792496 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.792775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.792536 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.792775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.792584 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2xg\" (UniqueName: \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.792775 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:38.792645 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-serving-cert: secret "maas-api-e2e-worker-w5-c7d477-serving-cert" not found Oct 30 21:34:38.792775 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:34:38.792731 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls podName:c9e1b468-709d-4e9e-b975-2d732b6bdc40 nodeName:}" failed. No retries permitted until 2026-10-30 21:34:39.292709329 +0000 UTC m=+703.966755862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls") pod "maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" (UID: "c9e1b468-709d-4e9e-b975-2d732b6bdc40") : secret "maas-api-e2e-worker-w5-c7d477-serving-cert" not found Oct 30 21:34:38.793384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.793357 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.795203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.795183 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:38.876651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:38.876617 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2xg\" (UniqueName: \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:39.297141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:39.297050 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:39.299593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:39.299570 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") pod \"maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:39.524837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:39.524803 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:39.899393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:39.899257 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 21:34:39.901715 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:34:39.901683 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e1b468_709d_4e9e_b975_2d732b6bdc40.slice/crio-e072773c0d327ed241253cf8a3bff2e5758f1a01a0804dbd47c607a5dcb00e25 WatchSource:0}: Error finding container e072773c0d327ed241253cf8a3bff2e5758f1a01a0804dbd47c607a5dcb00e25: Status 404 returned error can't find the container with id e072773c0d327ed241253cf8a3bff2e5758f1a01a0804dbd47c607a5dcb00e25 Oct 30 21:34:40.451698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:40.451665 4525 generic.go:356] "Generic (PLEG): container finished" podID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerID="d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df" exitCode=1 Oct 30 21:34:40.451900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:40.451741 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerDied","Data":"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df"} Oct 30 21:34:40.452179 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:40.452151 4525 scope.go:122] "RemoveContainer" containerID="d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df" Oct 30 21:34:40.452864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:40.452846 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" event={"ID":"c9e1b468-709d-4e9e-b975-2d732b6bdc40","Type":"ContainerStarted","Data":"e072773c0d327ed241253cf8a3bff2e5758f1a01a0804dbd47c607a5dcb00e25"} Oct 30 21:34:40.854891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:40.854855 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:34:41.459350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:41.459318 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerStarted","Data":"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049"} Oct 30 21:34:41.459808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:41.459547 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:41.460950 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:41.460932 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" event={"ID":"c9e1b468-709d-4e9e-b975-2d732b6bdc40","Type":"ContainerStarted","Data":"1eb56e8ee69be5e37b15a72d3afb9533946cb62b4a212d84936b031062fc25d8"} Oct 30 21:34:41.461067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:41.461051 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:41.711829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:41.711771 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" podStartSLOduration=2.9244129880000003 podStartE2EDuration="3.711751977s" totalImagesPullingTime="787.338989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:41.71007081 +0000 UTC m=+706.384117347" watchObservedRunningTime="2026-10-30 21:34:41.711751977 +0000 UTC m=+706.385798515" Oct 30 21:34:42.441417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:42.441389 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 21:34:45.427206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:45.427179 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 21:34:47.473128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:47.473101 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 21:34:47.473578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:47.473160 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 21:34:49.840055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:34:49.840020 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:35:00.557073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.557039 4525 generic.go:356] "Generic (PLEG): container finished" podID="419137d2-3574-453c-b867-90064a172154" containerID="f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a" exitCode=0 Oct 30 21:35:00.557504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.557115 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerDied","Data":"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a"} Oct 30 21:35:00.557504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.557479 4525 scope.go:122] "RemoveContainer" containerID="f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a" Oct 30 21:35:00.558681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.558655 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:00.558736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.558720 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:00.558766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:00.558751 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:01.564277 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:01.564236 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerStarted","Data":"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6"} Oct 30 21:35:02.152641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:02.152228 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:35:02.152641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:02.152370 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:35:02.152641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:02.152420 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:35:03.151067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:03.151026 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:35:03.151717 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:03.151682 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:35:04.149635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:04.148081 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:35:04.149635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:04.148147 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:35:05.152633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:05.152576 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:35:05.153137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:05.152669 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:35:05.350752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:05.350714 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:35:05.396274 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:35:05.396243 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:35:06.151102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:06.151062 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:35:06.151754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:06.151722 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 21:35:07.152093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:07.152065 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:35:07.153130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:07.153109 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:35:08.759936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:08.759901 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:35:11.870679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.870650 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 21:35:11.874178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.874162 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:11.940818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.940774 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 21:35:11.987191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.987151 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 21:35:11.988891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.988870 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-91874e-dockercfg-6njjh\"" Oct 30 21:35:11.989033 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.988932 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-91874e\"" Oct 30 21:35:11.989363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.989343 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:11.989405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.989395 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9v9f\" (UniqueName: \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:11.991045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:11.991032 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.059267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.059165 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 21:35:12.090458 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.090422 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:12.090679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.090506 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v9f\" (UniqueName: \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:12.090679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.090587 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfcl\" (UniqueName: \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.090679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.090658 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.091208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.091187 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:12.157804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.157721 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9v9f\" (UniqueName: \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\") pod \"payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:12.186667 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.186632 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:12.192070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.192033 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfcl\" (UniqueName: \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.192198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.192082 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.192867 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.192849 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.260513 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.260482 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfcl\" (UniqueName: \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\") pod \"payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.303751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.303716 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:12.583374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.583341 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 21:35:12.585130 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:35:12.585103 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf53135_9a0f_4c4c_aa43_8c4452773a20.slice/crio-e249bba9569403ce0893e5a3b1611ff401e98b10a474b2e0465a82c86ce40cff WatchSource:0}: Error finding container e249bba9569403ce0893e5a3b1611ff401e98b10a474b2e0465a82c86ce40cff: Status 404 returned error can't find the container with id e249bba9569403ce0893e5a3b1611ff401e98b10a474b2e0465a82c86ce40cff Oct 30 21:35:12.622556 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.622524 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" event={"ID":"bdf53135-9a0f-4c4c-aa43-8c4452773a20","Type":"ContainerStarted","Data":"e249bba9569403ce0893e5a3b1611ff401e98b10a474b2e0465a82c86ce40cff"} Oct 30 21:35:12.714653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:12.714577 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 21:35:12.714777 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:35:12.714756 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcead6166_5968_4180_bad3_6cc47ceec634.slice/crio-3a054c12402b63ab0f2d899d04a3745b3d081d7b4b195eb5357095fcfd248917 WatchSource:0}: Error finding container 3a054c12402b63ab0f2d899d04a3745b3d081d7b4b195eb5357095fcfd248917: Status 404 returned error can't find the container with id 3a054c12402b63ab0f2d899d04a3745b3d081d7b4b195eb5357095fcfd248917 Oct 30 21:35:13.644647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.635197 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" event={"ID":"cead6166-5968-4180-bad3-6cc47ceec634","Type":"ContainerStarted","Data":"c7bbc84ac2b9cb435fe800f4752c51f3954cc87c7cb84d1525cce7e932c936a6"} Oct 30 21:35:13.644647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.635264 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" event={"ID":"cead6166-5968-4180-bad3-6cc47ceec634","Type":"ContainerStarted","Data":"3a054c12402b63ab0f2d899d04a3745b3d081d7b4b195eb5357095fcfd248917"} Oct 30 21:35:13.644647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.636186 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:13.654764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.654700 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" event={"ID":"bdf53135-9a0f-4c4c-aa43-8c4452773a20","Type":"ContainerStarted","Data":"2d7539201cef2539761284be81967b1fa4908b24d9dbd3a53d28e860a88cd4b7"} Oct 30 21:35:13.654958 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.654857 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:13.756530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:13.756476 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" podStartSLOduration=2.756458206 podStartE2EDuration="2.756458206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:13.754239589 +0000 UTC m=+738.428286125" watchObservedRunningTime="2026-10-30 21:35:13.756458206 +0000 UTC m=+738.430504746" Oct 30 21:35:17.626288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:17.626225 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" podStartSLOduration=6.626205595 podStartE2EDuration="6.626205595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:13.883509884 +0000 UTC m=+738.557556442" watchObservedRunningTime="2026-10-30 21:35:17.626205595 +0000 UTC m=+742.300252136" Oct 30 21:35:17.628343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:17.628317 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:35:18.686313 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.686276 4525 generic.go:356] "Generic (PLEG): container finished" podID="8869ab31-ec35-487b-912f-76c5a07430ac" containerID="2d6b9e999940bfb14a2756f4dc56ba0c731af187bd701b4f4769c71627577614" exitCode=0 Oct 30 21:35:18.686768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.686343 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerDied","Data":"2d6b9e999940bfb14a2756f4dc56ba0c731af187bd701b4f4769c71627577614"} Oct 30 21:35:18.686768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.686756 4525 scope.go:122] "RemoveContainer" containerID="2d6b9e999940bfb14a2756f4dc56ba0c731af187bd701b4f4769c71627577614" Oct 30 21:35:18.688010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.687981 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:18.688097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.688041 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:18.688097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:18.688076 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:19.693910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:19.693874 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerStarted","Data":"6d3e71d3dde256acf4d5fcc4635da1323cc6eacf1e5e0bb448527a2439e118c9"} Oct 30 21:35:20.347780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:20.345738 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:35:20.475966 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:20.475932 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:35:20.477269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:20.477248 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:35:20.477332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:20.477309 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:35:21.476923 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:21.476891 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:35:21.477345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:21.476943 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:35:22.476650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:22.476250 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:35:22.476650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:22.476318 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:35:23.476542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:23.476511 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:35:23.476974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:23.476568 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:35:24.476987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:24.476951 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:35:24.477397 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:24.477005 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 21:35:24.665942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:24.665913 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 21:35:25.480442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:25.480400 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:35:25.481584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:25.481566 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 21:35:25.671286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:25.671255 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 21:35:35.377817 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.377767 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:35:35.378357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.377832 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.397838 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:35:35.397803 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:35:35.802996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.802898 4525 generic.go:356] "Generic (PLEG): container finished" podID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerID="fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f" exitCode=0 Oct 30 21:35:35.802996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.802958 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerDied","Data":"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f"} Oct 30 21:35:35.803416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.803399 4525 scope.go:122] "RemoveContainer" containerID="fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f" Oct 30 21:35:35.804781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.804753 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:35.804851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.804818 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:35.804892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:35.804849 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:35:36.401211 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:36.401118 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:35:36.810868 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:36.810781 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerStarted","Data":"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd"} Oct 30 21:35:37.135056 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:37.135023 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:35:37.135433 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:37.135340 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:35:37.138739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:37.138701 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:35:38.141083 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:38.141043 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:35:38.141647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:38.141128 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:35:39.136068 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:39.136026 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:35:39.136369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:39.136335 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:35:40.140634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:40.139079 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:35:40.140634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:40.139153 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:35:41.147637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:41.142061 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:35:41.147637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:41.142131 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:35:42.137787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:42.137751 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:35:42.138076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:42.138048 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:35:42.138819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:42.138802 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 21:35:57.312971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:35:57.312937 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:36:05.396537 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:36:05.396500 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:36:12.374154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:12.374118 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:36:26.859759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.859720 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 21:36:26.863864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.863838 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:26.968635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.968575 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"default-dockercfg-h647m\"" Oct 30 21:36:26.968896 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.968875 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"openshift-service-ca.crt\"" Oct 30 21:36:26.968981 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.968902 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"kube-root-ca.crt\"" Oct 30 21:36:26.971034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:26.971011 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000598 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000666 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000700 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhdmd\" (UniqueName: \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000779 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000830 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.001073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.000901 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.028849 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.026980 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d68156\"/\"premium-w3-d68156-kserve-self-signed-certs\"" Oct 30 21:36:27.102217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102186 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102232 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102275 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102321 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102338 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102369 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhdmd\" (UniqueName: \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102750 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102761 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.102863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.102828 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.104812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.104792 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.105034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.105013 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.164220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.164183 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhdmd\" (UniqueName: \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\") pod \"premium-w3-d68156-kserve-6865c485-jnm7d\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.190072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.190032 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:27.583344 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:27.583318 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 21:36:27.585421 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:36:27.585390 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e263cd1_b190_49ec_8564_f0f5171d8a12.slice/crio-0d3335df9731d5eeec930181304a99495bd678aad9e6c7a78d66a3a8696e2aa1 WatchSource:0}: Error finding container 0d3335df9731d5eeec930181304a99495bd678aad9e6c7a78d66a3a8696e2aa1: Status 404 returned error can't find the container with id 0d3335df9731d5eeec930181304a99495bd678aad9e6c7a78d66a3a8696e2aa1 Oct 30 21:36:28.079135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:28.079083 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" event={"ID":"7e263cd1-b190-49ec-8564-f0f5171d8a12","Type":"ContainerStarted","Data":"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e"} Oct 30 21:36:28.079135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:28.079139 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" event={"ID":"7e263cd1-b190-49ec-8564-f0f5171d8a12","Type":"ContainerStarted","Data":"0d3335df9731d5eeec930181304a99495bd678aad9e6c7a78d66a3a8696e2aa1"} Oct 30 21:36:28.191010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:28.190968 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:28.230076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:28.230015 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podStartSLOduration=2.230000656 podStartE2EDuration="2.230000656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:28.228811337 +0000 UTC m=+812.902857873" watchObservedRunningTime="2026-10-30 21:36:28.230000656 +0000 UTC m=+812.904047243" Oct 30 21:36:28.633279 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:28.633245 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:36:30.200454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.200415 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 21:36:30.204439 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.204406 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.306893 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.306862 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"kube-root-ca.crt\"" Oct 30 21:36:30.307381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.307352 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"premium-w4-1de75a-kserve-self-signed-certs\"" Oct 30 21:36:30.307508 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.307456 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"default-dockercfg-tv786\"" Oct 30 21:36:30.309590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.309566 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 21:36:30.335806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.335742 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xsb\" (UniqueName: \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.336363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.336309 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.336691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.336670 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.336963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.336941 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.337118 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.337104 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.346981 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.345822 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.358262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.358231 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"openshift-service-ca.crt\"" Oct 30 21:36:30.446650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446585 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xsb\" (UniqueName: \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.446825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446685 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.446825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446715 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.446825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446738 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.446825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446751 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.446825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.446794 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.447189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.447167 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.447252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.447187 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.447289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.447248 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.449306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.449278 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.449449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.449431 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.509107 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.509032 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xsb\" (UniqueName: \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\") pod \"premium-w4-1de75a-kserve-6598c87747-5jd4p\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.516864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.516833 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:30.916733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:30.916706 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 21:36:30.919143 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:36:30.919117 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3589c229_1f7b_481f_a040_0e14a166b7ee.slice/crio-27905bc545ddce6de73010e45885c35ea959d320a10cf4dc808a2dd9b217e5f0 WatchSource:0}: Error finding container 27905bc545ddce6de73010e45885c35ea959d320a10cf4dc808a2dd9b217e5f0: Status 404 returned error can't find the container with id 27905bc545ddce6de73010e45885c35ea959d320a10cf4dc808a2dd9b217e5f0 Oct 30 21:36:31.098856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:31.098820 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" event={"ID":"3589c229-1f7b-481f-a040-0e14a166b7ee","Type":"ContainerStarted","Data":"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df"} Oct 30 21:36:31.099068 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:31.098867 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" event={"ID":"3589c229-1f7b-481f-a040-0e14a166b7ee","Type":"ContainerStarted","Data":"27905bc545ddce6de73010e45885c35ea959d320a10cf4dc808a2dd9b217e5f0"} Oct 30 21:36:31.218883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:31.218771 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podStartSLOduration=1.218755314 podStartE2EDuration="1.218755314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:31.216837194 +0000 UTC m=+815.890883733" watchObservedRunningTime="2026-10-30 21:36:31.218755314 +0000 UTC m=+815.892801850" Oct 30 21:36:31.517725 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:31.517643 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:35.397200 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:36:35.397162 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:36:37.190855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:37.190818 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:37.203349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:37.203321 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:38.159906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:38.159872 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:36:40.517222 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:40.517185 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:40.529861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:40.529837 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:41.167476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:41.167447 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:36:49.974718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:36:49.974680 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:37:03.914320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:03.914287 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:37:03.919358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:03.919332 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:03.972334 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:03.972300 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d68156-openshift-default-dockercfg-wslcg\"" Oct 30 21:37:04.009841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.009813 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpdj\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.009970 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.009870 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010009 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.009988 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010059 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010025 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010100 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010171 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010138 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010216 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010290 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.010339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.010323 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.029407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.029371 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:37:04.111075 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111042 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpdj\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111104 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111125 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111144 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111178 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111208 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111263 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111308 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111348 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111706 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111657 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111774 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111791 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.111940 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.111917 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.112054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.112035 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.113756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.113735 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.113979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.113962 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.173378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.173311 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.173513 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.173437 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpdj\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.262789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.262751 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:04.693690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.693664 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:37:04.695743 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:37:04.695717 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaa84ab_14e9_4977_96e5_eda7b4a3b955.slice/crio-8b09d46d8c7aece033ae1da6fbd39e96a495c4520c03d873e4eff0c1b05a9fc0 WatchSource:0}: Error finding container 8b09d46d8c7aece033ae1da6fbd39e96a495c4520c03d873e4eff0c1b05a9fc0: Status 404 returned error can't find the container with id 8b09d46d8c7aece033ae1da6fbd39e96a495c4520c03d873e4eff0c1b05a9fc0 Oct 30 21:37:04.698272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.698239 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:04.698335 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.698312 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:04.698384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:04.698340 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:05.278956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:05.278918 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" event={"ID":"3aaa84ab-14e9-4977-96e5-eda7b4a3b955","Type":"ContainerStarted","Data":"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361"} Oct 30 21:37:05.279441 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:05.278968 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" event={"ID":"3aaa84ab-14e9-4977-96e5-eda7b4a3b955","Type":"ContainerStarted","Data":"8b09d46d8c7aece033ae1da6fbd39e96a495c4520c03d873e4eff0c1b05a9fc0"} Oct 30 21:37:05.397479 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:37:05.397442 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:37:05.422008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:05.421961 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podStartSLOduration=2.421945937 podStartE2EDuration="2.421945937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:05.419718229 +0000 UTC m=+850.093764776" watchObservedRunningTime="2026-10-30 21:37:05.421945937 +0000 UTC m=+850.095992472" Oct 30 21:37:06.263111 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:06.263020 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:06.264213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:06.264194 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:37:06.264283 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:06.264248 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 21:37:07.263995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:07.263955 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:37:07.264450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:07.264023 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 21:37:08.264230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:08.264190 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:37:08.264689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:08.264261 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 21:37:09.263235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:09.263188 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:37:09.263421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:09.263256 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 21:37:10.263905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:10.263863 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:37:10.264309 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:10.263924 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 21:37:11.438367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:11.438332 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:11.438882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:11.438780 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:11.439711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:11.439683 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:37:12.058925 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:12.058891 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:37:35.395799 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:37:35.395761 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:37:48.244659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.244625 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:37:48.248273 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.248252 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299465 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299512 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299582 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299668 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299693 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299720 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwdx\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299867 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.299997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.299906 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.300061 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.300044 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.301289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.301268 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1de75a-openshift-default-dockercfg-rtvkc\"" Oct 30 21:37:48.354144 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.354112 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:37:48.401011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.400968 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401044 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401077 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401107 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwdx\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401156 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401179 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401315 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401352 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401389 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401664 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401832 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401967 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401902 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.401967 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.401922 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.402359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.402338 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.403600 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.403580 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.404012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.403992 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.462684 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.462650 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.462831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.462786 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwdx\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.561696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.561573 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:48.992295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.992267 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:37:48.993924 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:37:48.993898 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d25c6f8_5a41_4a27_bee2_4181836d09ad.slice/crio-9f27a2f0a8951242187477599afb9a7f6819901f89ce86716168caf6056766d2 WatchSource:0}: Error finding container 9f27a2f0a8951242187477599afb9a7f6819901f89ce86716168caf6056766d2: Status 404 returned error can't find the container with id 9f27a2f0a8951242187477599afb9a7f6819901f89ce86716168caf6056766d2 Oct 30 21:37:48.996092 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.996058 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:48.996144 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.996125 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:48.996188 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:48.996155 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:37:49.512346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:49.512310 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" event={"ID":"5d25c6f8-5a41-4a27-bee2-4181836d09ad","Type":"ContainerStarted","Data":"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886"} Oct 30 21:37:49.512822 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:49.512365 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" event={"ID":"5d25c6f8-5a41-4a27-bee2-4181836d09ad","Type":"ContainerStarted","Data":"9f27a2f0a8951242187477599afb9a7f6819901f89ce86716168caf6056766d2"} Oct 30 21:37:49.561915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:49.561874 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:49.686407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:49.686355 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podStartSLOduration=1.686339625 podStartE2EDuration="1.686339625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:49.683076312 +0000 UTC m=+894.357122848" watchObservedRunningTime="2026-10-30 21:37:49.686339625 +0000 UTC m=+894.360386161" Oct 30 21:37:49.842692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:49.829671 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:37:50.563711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:50.563676 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:37:50.564176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:50.563737 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 21:37:51.562674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:51.562641 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:37:51.562859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:51.562690 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 21:37:52.562003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:52.561971 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:37:52.562428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:52.562020 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 21:37:53.562855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:53.562821 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:37:53.563349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:53.562874 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 21:37:54.562809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:54.562768 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:37:54.563003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:54.562834 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 21:37:55.814575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:55.814544 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:55.815071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:55.814960 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:37:55.815948 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:37:55.815931 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:38:05.396519 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:38:05.396476 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:38:13.715265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:13.715230 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:38:13.715779 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:13.715469 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" containerID="cri-o://d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b" gracePeriod=30 Oct 30 21:38:18.974073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:18.974044 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.156977 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157043 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157091 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157151 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157185 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157232 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157264 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157351 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.157421 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\") pod \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\" (UID: \"72b14672-aa4e-4179-a9c4-3c01024fcb3a\") " Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.158039 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.158308 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.158515 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:19.159561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.159279 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:19.161136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.161107 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:19.161136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.161124 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:19.161356 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.161338 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:19.162400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.162363 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "kube-api-access-z68cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:19.163287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.163263 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy" pod "72b14672-aa4e-4179-a9c4-3c01024fcb3a" (UID: "72b14672-aa4e-4179-a9c4-3c01024fcb3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:19.258444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258406 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258439 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68cs\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-kube-api-access-z68cs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258454 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258465 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258477 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258488 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258499 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b14672-aa4e-4179-a9c4-3c01024fcb3a-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258509 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.258726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.258521 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b14672-aa4e-4179-a9c4-3c01024fcb3a-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:19.692452 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.692406 4525 generic.go:356] "Generic (PLEG): container finished" podID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerID="d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b" exitCode=0 Oct 30 21:38:19.692675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.692464 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" event={"ID":"72b14672-aa4e-4179-a9c4-3c01024fcb3a","Type":"ContainerDied","Data":"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b"} Oct 30 21:38:19.692675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.692489 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" Oct 30 21:38:19.692675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.692515 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg" event={"ID":"72b14672-aa4e-4179-a9c4-3c01024fcb3a","Type":"ContainerDied","Data":"0cb1194aac6f2d69ddff8bdde8472d3af5fd6534cb38fefbca2076b1634b52dc"} Oct 30 21:38:19.692675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.692539 4525 scope.go:122] "RemoveContainer" containerID="d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b" Oct 30 21:38:19.702623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.702574 4525 scope.go:122] "RemoveContainer" containerID="d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b" Oct 30 21:38:19.702977 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:38:19.702949 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b\": container with ID starting with d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b not found: ID does not exist" containerID="d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b" Oct 30 21:38:19.703067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.702986 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b"} err="failed to get container status \"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b\": rpc error: code = NotFound desc = could not find container \"d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b\": container with ID starting with d5f9a6fa1ff0e6aa536d69f9c8b27ca73737b67158ca2b9e2464eb556864df4b not found: ID does not exist" Oct 30 21:38:19.863191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.863096 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:38:19.871154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.871115 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-52mqg"] Oct 30 21:38:19.960991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.960958 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 21:38:19.962626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.962578 4525 cpu_manager.go:414] "containerMap: removing container" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" Oct 30 21:38:19.962750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.962665 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" Oct 30 21:38:19.963643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.962970 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" containerName="istio-proxy" Oct 30 21:38:19.973864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:19.973836 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.032156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.030757 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"openshift-service-ca.crt\"" Oct 30 21:38:20.032156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.031521 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"default-dockercfg-pm68c\"" Oct 30 21:38:20.032156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.031672 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"premium-w1-6ea374-kserve-self-signed-certs\"" Oct 30 21:38:20.032156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.031735 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6ea374\"/\"kube-root-ca.crt\"" Oct 30 21:38:20.036631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.034287 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 21:38:20.073469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073435 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.073709 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073506 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdsh\" (UniqueName: \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.073709 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073692 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.073870 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073849 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.073980 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073900 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.074062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.073998 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175130 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175176 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175192 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175226 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175276 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdsh\" (UniqueName: \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175429 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175582 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175692 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.175866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.175849 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.177809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.177773 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.178132 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.178107 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.240585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.240547 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdsh\" (UniqueName: \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\") pod \"premium-w1-6ea374-kserve-f88b69587-78wb4\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.286194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.286156 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:20.665465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.665442 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 21:38:20.666819 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:38:20.666787 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27bd8c87_1929_45e7_a617_18015b60bcbd.slice/crio-259d6ee067ed495e9df81028249e07f64969e4514abd89cfc1e7e55b5d133b33 WatchSource:0}: Error finding container 259d6ee067ed495e9df81028249e07f64969e4514abd89cfc1e7e55b5d133b33: Status 404 returned error can't find the container with id 259d6ee067ed495e9df81028249e07f64969e4514abd89cfc1e7e55b5d133b33 Oct 30 21:38:20.700832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:20.700793 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" event={"ID":"27bd8c87-1929-45e7-a617-18015b60bcbd","Type":"ContainerStarted","Data":"259d6ee067ed495e9df81028249e07f64969e4514abd89cfc1e7e55b5d133b33"} Oct 30 21:38:21.445088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:21.445048 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:38:21.473491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:21.473452 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b14672-aa4e-4179-a9c4-3c01024fcb3a" path="/var/lib/kubelet/pods/72b14672-aa4e-4179-a9c4-3c01024fcb3a/volumes" Oct 30 21:38:21.709945 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:21.709846 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" event={"ID":"27bd8c87-1929-45e7-a617-18015b60bcbd","Type":"ContainerStarted","Data":"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875"} Oct 30 21:38:21.830517 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:21.830466 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podStartSLOduration=2.830450772 podStartE2EDuration="2.830450772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:21.829048199 +0000 UTC m=+926.503094747" watchObservedRunningTime="2026-10-30 21:38:21.830450772 +0000 UTC m=+926.504497308" Oct 30 21:38:22.286510 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:22.286456 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:30.287193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:30.287149 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:30.300525 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:30.300497 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:30.770329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:30.770299 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:38:35.370679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.370637 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 21:38:35.377496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.377466 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.405451 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:38:35.405415 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:38:35.430104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.430074 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 21:38:35.434404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434370 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.434562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434446 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.434649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434625 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.434727 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434708 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.434783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434740 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.434844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.434803 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pld2\" (UniqueName: \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.476945 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.476912 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"openshift-service-ca.crt\"" Oct 30 21:38:35.479701 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.479678 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"kube-root-ca.crt\"" Oct 30 21:38:35.479701 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.479680 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"premium-w2-91874e-kserve-self-signed-certs\"" Oct 30 21:38:35.479701 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.479705 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-91874e\"/\"default-dockercfg-wsbzn\"" Oct 30 21:38:35.543108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543068 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.543324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543171 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.543324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543268 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.543456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543331 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.543456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543352 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.543456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.543391 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pld2\" (UniqueName: \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.544094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.544064 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.544221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.544174 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.544221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.544207 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.547404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.547375 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.547923 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.547901 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.605284 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.605251 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pld2\" (UniqueName: \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\") pod \"premium-w2-91874e-kserve-575bcd577b-tfhsk\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:35.691172 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:35.691078 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:36.093959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:36.093930 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 21:38:36.095206 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:38:36.095175 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded5bd236_ca76_47db_8d20_64df7cdd4e5e.slice/crio-9ba4b18e445631f5d685468039e7850e83286e556b01d5f71cd803c944663950 WatchSource:0}: Error finding container 9ba4b18e445631f5d685468039e7850e83286e556b01d5f71cd803c944663950: Status 404 returned error can't find the container with id 9ba4b18e445631f5d685468039e7850e83286e556b01d5f71cd803c944663950 Oct 30 21:38:36.800562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:36.800527 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" event={"ID":"ed5bd236-ca76-47db-8d20-64df7cdd4e5e","Type":"ContainerStarted","Data":"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934"} Oct 30 21:38:36.800562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:36.800570 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" event={"ID":"ed5bd236-ca76-47db-8d20-64df7cdd4e5e","Type":"ContainerStarted","Data":"9ba4b18e445631f5d685468039e7850e83286e556b01d5f71cd803c944663950"} Oct 30 21:38:36.919052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:36.918995 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podStartSLOduration=1.918977816 podStartE2EDuration="1.918977816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:36.917220541 +0000 UTC m=+941.591267088" watchObservedRunningTime="2026-10-30 21:38:36.918977816 +0000 UTC m=+941.593024352" Oct 30 21:38:37.691962 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:37.691927 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:43.224708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:43.224677 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:38:45.691749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:45.691719 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:45.704844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:45.704818 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:45.863341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:45.863311 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:38:58.944880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:58.944837 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:38:58.949819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:58.949794 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.002427 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.002392 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91874e-openshift-default-dockercfg-zjpd2\"" Oct 30 21:38:59.026405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026364 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026445 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026546 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026641 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026690 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026811 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026789 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026849 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026825 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.026905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026860 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.027008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.026988 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wctf\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.054708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.054673 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:38:59.127597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127564 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.127597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127597 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.127869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127644 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.127869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127680 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wctf\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.127869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127747 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.127869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127778 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.127951 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128001 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128046 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128053 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128094 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128151 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128287 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.128881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.128860 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.130254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.130230 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.130347 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.130330 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.192744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.192713 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.193391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.193365 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wctf\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.263643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.263526 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:38:59.694850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.694822 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:38:59.696039 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:38:59.696017 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae48d365_ae0d_45ff_9b07_761218945f63.slice/crio-1dd1eda9595bec43872758e9e23ee370e18148f38716471e6c39053a740c1f43 WatchSource:0}: Error finding container 1dd1eda9595bec43872758e9e23ee370e18148f38716471e6c39053a740c1f43: Status 404 returned error can't find the container with id 1dd1eda9595bec43872758e9e23ee370e18148f38716471e6c39053a740c1f43 Oct 30 21:38:59.699333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.699298 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:38:59.699419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.699369 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:38:59.699419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.699399 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:38:59.927008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.926970 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" event={"ID":"ae48d365-ae0d-45ff-9b07-761218945f63","Type":"ContainerStarted","Data":"e2b34db45369dd29084539de63de3b602e69578e58da6f99f6b07c83bf38e20f"} Oct 30 21:38:59.927214 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:38:59.927037 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" event={"ID":"ae48d365-ae0d-45ff-9b07-761218945f63","Type":"ContainerStarted","Data":"1dd1eda9595bec43872758e9e23ee370e18148f38716471e6c39053a740c1f43"} Oct 30 21:39:00.072707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:00.072547 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podStartSLOduration=2.072524166 podStartE2EDuration="2.072524166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:00.070846784 +0000 UTC m=+964.744893357" watchObservedRunningTime="2026-10-30 21:39:00.072524166 +0000 UTC m=+964.746570701" Oct 30 21:39:00.264546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:00.264508 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:39:00.266075 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:00.266050 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:00.266206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:00.266100 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:01.264492 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:01.264457 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:01.265101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:01.264510 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:02.264305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:02.264269 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:02.264491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:02.264340 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:03.264756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:03.264717 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:03.265152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:03.264782 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:04.264400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:04.264358 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:04.264660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:04.264418 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:05.264224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:05.264181 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:39:05.264677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:05.264266 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 21:39:05.397881 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:39:05.397847 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:39:06.461435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:06.461406 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:39:06.462021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:06.461955 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:39:06.463141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:06.463118 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:39:10.925522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:10.925476 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:39:35.250787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.250746 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 21:39:35.261929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.261891 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268489 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268459 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stp4b\" (UniqueName: \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268510 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268639 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268667 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268698 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.268851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.268756 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.322655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.321681 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 21:39:35.360884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.360826 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"kube-root-ca.crt\"" Oct 30 21:39:35.360884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.360826 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"openshift-service-ca.crt\"" Oct 30 21:39:35.369701 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.369667 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.370047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.369972 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.370310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.370287 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.370895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.370636 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.370895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.370788 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.371568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.371083 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stp4b\" (UniqueName: \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.371568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.371148 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.371869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.371728 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.372091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.372072 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.372262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.372239 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"distinct-w6-69e4b2-kserve-self-signed-certs\"" Oct 30 21:39:35.372492 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.372476 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"default-dockercfg-rnhwl\"" Oct 30 21:39:35.375201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.375154 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.385372 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.385345 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.447917 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.447883 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stp4b\" (UniqueName: \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\") pod \"distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:35.450328 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:39:35.450290 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:39:35.577861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:35.577764 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:36.049231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:36.049205 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 21:39:36.051319 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:39:36.051285 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ba82b0_c5c1_48bc_be00_0ca072a80ef5.slice/crio-4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46 WatchSource:0}: Error finding container 4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46: Status 404 returned error can't find the container with id 4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46 Oct 30 21:39:36.153984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:36.153943 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" event={"ID":"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5","Type":"ContainerStarted","Data":"1f53bd9f71ac5d98ba60e7f3c56f5f6dbad7840db85d0bcfd92789c6e13977b7"} Oct 30 21:39:36.154182 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:36.153998 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" event={"ID":"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5","Type":"ContainerStarted","Data":"4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46"} Oct 30 21:39:36.273369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:36.273303 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podStartSLOduration=1.273281802 podStartE2EDuration="1.273281802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:36.271356831 +0000 UTC m=+1000.945403368" watchObservedRunningTime="2026-10-30 21:39:36.273281802 +0000 UTC m=+1000.947328345" Oct 30 21:39:36.580119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:36.578844 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:44.459395 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.459363 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:39:44.463157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.463139 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.560803 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.560766 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:39:44.580042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580004 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580044 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580299 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580061 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580299 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580085 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580299 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580208 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580411 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jh9v\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580533 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580476 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580588 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580572 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.580731 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.580711 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.681996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.681960 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jh9v\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682016 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682047 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682099 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682148 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682178 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682193 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682223 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682249 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682581 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682772 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.682802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.682778 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.683034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.683017 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.683157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.683133 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.684834 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.684812 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.685246 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.685220 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.744055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.743970 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.744206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.744060 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jh9v\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-kt59r\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:44.776373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:44.776338 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:45.205122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.205094 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:39:45.206810 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:39:45.206779 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53b560f_0727_4b17_8ee6_8bd491c102c6.slice/crio-9e55b8c89e28617b49f0fed964895dc886ecacb4e8899af15eb51b88744ca450 WatchSource:0}: Error finding container 9e55b8c89e28617b49f0fed964895dc886ecacb4e8899af15eb51b88744ca450: Status 404 returned error can't find the container with id 9e55b8c89e28617b49f0fed964895dc886ecacb4e8899af15eb51b88744ca450 Oct 30 21:39:45.210883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.210845 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:45.210999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.210917 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:45.210999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.210957 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:45.578627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.578519 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:45.597790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:45.597755 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:46.212365 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.212328 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" event={"ID":"f53b560f-0727-4b17-8ee6-8bd491c102c6","Type":"ContainerStarted","Data":"691023665c23579fd09af814e1c3e9a0d512b2425449eb5b77fbce5fde325e27"} Oct 30 21:39:46.212554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.212387 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" event={"ID":"f53b560f-0727-4b17-8ee6-8bd491c102c6","Type":"ContainerStarted","Data":"9e55b8c89e28617b49f0fed964895dc886ecacb4e8899af15eb51b88744ca450"} Oct 30 21:39:46.224551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.224520 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:39:46.459572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.459509 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podStartSLOduration=2.459493123 podStartE2EDuration="2.459493123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.341028415 +0000 UTC m=+1011.015074952" watchObservedRunningTime="2026-10-30 21:39:46.459493123 +0000 UTC m=+1011.133539659" Oct 30 21:39:46.777054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.777021 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:46.778678 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.778650 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.778800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:46.778713 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:47.776870 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:47.776831 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:47.777061 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:47.776893 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:48.777216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:48.777175 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:48.777660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:48.777244 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:49.777335 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.777301 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:49.777877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.777354 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:49.881915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.881884 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 21:39:49.885660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.885641 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.951633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.939453 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-69e4b2-openshift-default-dockercfg-qvwv2\"" Oct 30 21:39:49.971516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971484 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971521 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971681 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dll\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971745 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971787 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971888 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.971943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.971934 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.972024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.972009 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.972066 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.972055 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:49.991769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:49.991738 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 21:39:50.073206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073116 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073169 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073286 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073323 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073389 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dll\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073436 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073453 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073488 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073535 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073541 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073698 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073805 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.073898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.073847 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.074276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.074213 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.075925 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.075896 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.076103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.076085 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.155424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.155385 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.157325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.157112 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dll\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.198899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.198867 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:50.658494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.658444 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 21:39:50.662937 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:39:50.662907 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfb88d8_7d4e_436a_aaf5_79189fec1661.slice/crio-64f625e1fbff99d6f36e1bb757b09f3b84cfeb8b605b14eabe5b7037c9b90f4d WatchSource:0}: Error finding container 64f625e1fbff99d6f36e1bb757b09f3b84cfeb8b605b14eabe5b7037c9b90f4d: Status 404 returned error can't find the container with id 64f625e1fbff99d6f36e1bb757b09f3b84cfeb8b605b14eabe5b7037c9b90f4d Oct 30 21:39:50.665394 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.665363 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:50.665510 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.665441 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:50.665510 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.665470 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:39:50.778112 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.778078 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:50.778714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:50.778141 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:51.276349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:51.276308 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" event={"ID":"ccfb88d8-7d4e-436a-aaf5-79189fec1661","Type":"ContainerStarted","Data":"d87722582509fe0d7654b75fee04689e1f624820791c11892a211a08913de752"} Oct 30 21:39:51.276572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:51.276387 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" event={"ID":"ccfb88d8-7d4e-436a-aaf5-79189fec1661","Type":"ContainerStarted","Data":"64f625e1fbff99d6f36e1bb757b09f3b84cfeb8b605b14eabe5b7037c9b90f4d"} Oct 30 21:39:51.401870 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:51.401784 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podStartSLOduration=2.401767663 podStartE2EDuration="2.401767663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:51.401487945 +0000 UTC m=+1016.075534494" watchObservedRunningTime="2026-10-30 21:39:51.401767663 +0000 UTC m=+1016.075814201" Oct 30 21:39:51.776948 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:51.776914 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:51.777156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:51.776964 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:52.199657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:52.199593 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:52.201518 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:52.201490 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:52.201687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:52.201547 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:52.777762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:52.777718 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kt59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:39:52.777991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:52.777789 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 21:39:53.200057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:53.200020 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:53.200478 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:53.200090 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:54.006357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:54.006326 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:54.006766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:54.006726 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:54.007578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:54.007544 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:39:54.200204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:54.200171 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:54.200676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:54.200238 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:55.200001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:55.199967 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.200195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:55.200020 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:56.199653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:56.199597 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:56.200143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:56.199688 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:56.827832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:56.827798 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:39:57.199459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:57.199419 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.199832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:57.199800 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 21:39:58.971384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:58.971346 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:58.972562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:58.972502 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:39:58.973468 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:39:58.973444 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 21:40:05.347486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.347448 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:40:05.352520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.352499 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.396763 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:40:05.396724 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:40:05.455766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.455734 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:40:05.526358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526321 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526365 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526392 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526515 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd65f\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526571 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526671 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526693 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526714 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.526904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.526785 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628175 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628145 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628201 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628221 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628248 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628304 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628406 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628443 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628476 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628517 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd65f\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628709 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.628837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.628748 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.629156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.629126 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.629272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.629177 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.629319 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.629283 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.630898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.630872 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.631026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.631005 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.693695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.693663 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.694984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.694959 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd65f\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:05.967342 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:05.967245 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:06.407246 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:06.407210 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:40:06.415387 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:40:06.415328 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84218860_63ed_4fe2_9489_41bf1bd3dafc.slice/crio-22dfe60d5746fb966dc4c027b2db64d3fae16c2ae3f2b322285208c0c77a4f07 WatchSource:0}: Error finding container 22dfe60d5746fb966dc4c027b2db64d3fae16c2ae3f2b322285208c0c77a4f07: Status 404 returned error can't find the container with id 22dfe60d5746fb966dc4c027b2db64d3fae16c2ae3f2b322285208c0c77a4f07 Oct 30 21:40:06.420891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:06.419437 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:06.420891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:06.419527 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:06.420891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:06.419567 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:07.406964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.406925 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" event={"ID":"84218860-63ed-4fe2-9489-41bf1bd3dafc","Type":"ContainerStarted","Data":"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da"} Oct 30 21:40:07.407223 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.406999 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" event={"ID":"84218860-63ed-4fe2-9489-41bf1bd3dafc","Type":"ContainerStarted","Data":"22dfe60d5746fb966dc4c027b2db64d3fae16c2ae3f2b322285208c0c77a4f07"} Oct 30 21:40:07.545632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.531665 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podStartSLOduration=2.53164447 podStartE2EDuration="2.53164447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:07.529313364 +0000 UTC m=+1032.203359901" watchObservedRunningTime="2026-10-30 21:40:07.53164447 +0000 UTC m=+1032.205691007" Oct 30 21:40:07.967689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.967648 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:07.969464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.969438 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:07.969599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:07.969491 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:08.967956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:08.967914 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:08.968472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:08.967982 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:09.968517 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:09.968478 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:09.968921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:09.968541 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:10.214360 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:10.214327 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:40:10.970636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:10.968087 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.970636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:10.968146 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:11.969133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:11.969091 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.969480 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:11.969447 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:12.968453 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:12.968399 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.968892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:12.968483 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 21:40:13.973172 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:13.973143 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:13.973673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:13.973646 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:13.974577 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:13.974555 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:40:29.614819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:29.614790 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:40:34.595118 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.595085 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:40:34.600228 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.600208 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.702753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.702722 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:40:34.773424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773390 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773431 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773454 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773572 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773587 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773635 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773683 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773812 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.773875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.773839 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bhb\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874498 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874460 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874505 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874533 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874640 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874665 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874689 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874716 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874820 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.874989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.874850 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bhb\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.875497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.875464 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.875599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.875523 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.875599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.875570 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.875741 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.875700 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.876003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.875986 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.878077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.878044 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.878276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.878261 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.937037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.937010 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:34.937200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:34.937096 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bhb\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:35.215236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.215159 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:35.396101 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:40:35.396065 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:40:35.665289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.652759 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:40:35.665289 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:40:35.654427 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83e39ca_ed22_41a6_a062_b58c4b1a7fa7.slice/crio-de1cac3f3b6199dde79a6ccd3b464d3a4838800b0d6b4b79d4ed5582ebbeb34d WatchSource:0}: Error finding container de1cac3f3b6199dde79a6ccd3b464d3a4838800b0d6b4b79d4ed5582ebbeb34d: Status 404 returned error can't find the container with id de1cac3f3b6199dde79a6ccd3b464d3a4838800b0d6b4b79d4ed5582ebbeb34d Oct 30 21:40:35.665289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.656840 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:35.665289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.656908 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:35.665289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.656944 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:40:35.674544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:35.674516 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" event={"ID":"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7","Type":"ContainerStarted","Data":"de1cac3f3b6199dde79a6ccd3b464d3a4838800b0d6b4b79d4ed5582ebbeb34d"} Oct 30 21:40:36.681440 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:36.681399 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" event={"ID":"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7","Type":"ContainerStarted","Data":"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7"} Oct 30 21:40:36.911056 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:36.911000 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podStartSLOduration=2.910986191 podStartE2EDuration="2.910986191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:36.860033415 +0000 UTC m=+1061.534079949" watchObservedRunningTime="2026-10-30 21:40:36.910986191 +0000 UTC m=+1061.585032727" Oct 30 21:40:37.215468 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:37.215432 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:37.217110 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:37.217083 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:40:37.217262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:37.217133 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:40:38.216658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:38.216409 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:40:38.216658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:38.216470 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:40:39.216469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:39.216434 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:40:39.216700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:39.216494 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:40:40.215721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:40.215687 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:40:40.215913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:40.215751 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:40:41.216156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:41.216120 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:40:41.216680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:41.216194 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 21:40:42.407791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:42.407746 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:42.408815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:42.408788 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:42.409651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:42.409624 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:40:57.048166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:40:57.048131 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:41:05.396868 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:41:05.396822 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:41:31.268833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:31.268794 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:41:35.396904 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:41:35.396866 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:41:37.529904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.529814 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 21:41:37.534124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.534098 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.595048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.595014 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 21:41:37.640491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.640462 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-69e4b2\"/\"distinct-2-w6-69e4b2-kserve-self-signed-certs\"" Oct 30 21:41:37.707107 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.707064 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.707543 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.707510 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.707773 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.707757 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.708082 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.708067 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.708291 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.708235 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.708568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.708529 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtjz\" (UniqueName: \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.810943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.810855 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.810943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.810894 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.810943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.810925 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtjz\" (UniqueName: \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.810967 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.811024 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.811039 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.811339 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.811347 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.811498 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.811445 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.813556 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.813533 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.813717 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.813702 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:37.873097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:37.873063 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtjz\" (UniqueName: \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\") pod \"distinct-2-w6-69e4b2-kserve-648785df96-zcgl9\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:38.148796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:38.148767 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:38.525910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:38.525883 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 21:41:38.527484 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:41:38.527450 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaaed7af_6507_493a_867d_880a4c5f1395.slice/crio-90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86 WatchSource:0}: Error finding container 90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86: Status 404 returned error can't find the container with id 90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86 Oct 30 21:41:39.075130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:39.075095 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" event={"ID":"eaaed7af-6507-493a-867d-880a4c5f1395","Type":"ContainerStarted","Data":"4a0067a264ebfc0732de5ad83d142dae1b3233702f39c44f20e5902fba82e3de"} Oct 30 21:41:39.075547 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:39.075144 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" event={"ID":"eaaed7af-6507-493a-867d-880a4c5f1395","Type":"ContainerStarted","Data":"90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86"} Oct 30 21:41:39.149114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:39.149080 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:39.195761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:39.195702 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podStartSLOduration=2.195682533 podStartE2EDuration="2.195682533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:39.192398532 +0000 UTC m=+1123.866445069" watchObservedRunningTime="2026-10-30 21:41:39.195682533 +0000 UTC m=+1123.869729069" Oct 30 21:41:44.833586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.833550 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 21:41:44.837424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.837404 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895110 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895223 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895256 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895335 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpdm\" (UniqueName: \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895578 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.895660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.895641 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:44.942751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.942708 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"distinct-w4-1de75a-kserve-self-signed-certs\"" Oct 30 21:41:44.945129 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:44.945104 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 21:41:45.003637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.003574 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.004112 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.004084 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.010477 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.004818 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.011054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.010988 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpdm\" (UniqueName: \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.011916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.011886 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.012048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.011962 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.012048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.012013 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.012548 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.012512 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.012895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.012872 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.029790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.029743 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.032408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.031708 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.072961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.072928 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpdm\" (UniqueName: \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\") pod \"distinct-w4-1de75a-kserve-55b7749f4-xjlvl\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.158648 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.158589 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:45.609348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:45.609321 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 21:41:45.611088 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:41:45.611052 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35c60783_4694_47d1_8e7e_c44a206867dd.slice/crio-730b2bfb5ca7d026c6096fb99e31331f652d95a7abbca181d48231c2280a7ab8 WatchSource:0}: Error finding container 730b2bfb5ca7d026c6096fb99e31331f652d95a7abbca181d48231c2280a7ab8: Status 404 returned error can't find the container with id 730b2bfb5ca7d026c6096fb99e31331f652d95a7abbca181d48231c2280a7ab8 Oct 30 21:41:46.118682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:46.118640 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" event={"ID":"35c60783-4694-47d1-8e7e-c44a206867dd","Type":"ContainerStarted","Data":"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830"} Oct 30 21:41:46.119177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:46.118694 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" event={"ID":"35c60783-4694-47d1-8e7e-c44a206867dd","Type":"ContainerStarted","Data":"730b2bfb5ca7d026c6096fb99e31331f652d95a7abbca181d48231c2280a7ab8"} Oct 30 21:41:46.159215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:46.159187 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:46.294429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:46.294376 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podStartSLOduration=2.294361925 podStartE2EDuration="2.294361925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:46.291435457 +0000 UTC m=+1130.965481992" watchObservedRunningTime="2026-10-30 21:41:46.294361925 +0000 UTC m=+1130.968408461" Oct 30 21:41:48.149600 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:48.149571 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:48.162449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:48.162425 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:49.147180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:49.147149 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:41:55.159582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:55.159545 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:55.176081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:55.176059 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:41:55.187427 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:41:55.187401 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:42:00.391530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.391500 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 21:42:00.395695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.395670 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.498336 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.498295 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 21:42:00.537445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.537401 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.537819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.537786 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.537958 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.537869 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.537967 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.538072 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538146 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.538099 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.538148 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538296 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.538273 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn8q\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.538359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.538316 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639472 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639553 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639588 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639705 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639727 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639768 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639851 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn8q\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639879 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.639953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639923 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.640143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.639974 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.640143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.640095 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.640143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.640101 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.640303 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.640263 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.640544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.640518 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.642785 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.642730 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.642877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.642859 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.701063 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.701030 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn8q\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.701226 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.701084 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:00.709413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:00.709386 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:01.165644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:01.165618 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 21:42:01.166895 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:42:01.166848 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fcf030_0ddf_4cdc_96c5_ba642c790659.slice/crio-191e06ea498610b1efb661154477cc575b82bbc80d6b7d18bbeae129727ed637 WatchSource:0}: Error finding container 191e06ea498610b1efb661154477cc575b82bbc80d6b7d18bbeae129727ed637: Status 404 returned error can't find the container with id 191e06ea498610b1efb661154477cc575b82bbc80d6b7d18bbeae129727ed637 Oct 30 21:42:01.169806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:01.169767 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:42:01.169930 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:01.169859 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:42:01.169930 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:01.169919 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:42:01.205210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:01.205178 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" event={"ID":"f0fcf030-0ddf-4cdc-96c5-ba642c790659","Type":"ContainerStarted","Data":"191e06ea498610b1efb661154477cc575b82bbc80d6b7d18bbeae129727ed637"} Oct 30 21:42:02.212067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:02.212025 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" event={"ID":"f0fcf030-0ddf-4cdc-96c5-ba642c790659","Type":"ContainerStarted","Data":"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5"} Oct 30 21:42:02.385326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:02.385269 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podStartSLOduration=2.385255254 podStartE2EDuration="2.385255254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.383190836 +0000 UTC m=+1147.057237374" watchObservedRunningTime="2026-10-30 21:42:02.385255254 +0000 UTC m=+1147.059301870" Oct 30 21:42:02.710118 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:02.710089 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:02.711660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:02.711632 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:02.711807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:02.711684 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:03.709934 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:03.709899 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:03.710350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:03.709952 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:04.710702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:04.710668 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:04.711109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:04.710724 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:05.398499 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:42:05.398459 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:42:05.710293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:05.710205 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:05.710293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:05.710265 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:06.709853 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:06.709817 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:06.710337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:06.709880 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:07.710331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:07.710295 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:42:07.710748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:07.710383 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 21:42:08.714102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:08.714070 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:08.714583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:08.714359 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:08.715051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:08.715033 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 21:42:10.201055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:10.201023 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:42:35.397207 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:42:35.397170 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:42:40.708130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:40.708093 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:42:50.785000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.784966 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 21:42:50.789023 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.789001 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839107 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839263 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839380 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82q4f\" (UniqueName: \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839404 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839475 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839441 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.839511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.839503 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.842862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.842839 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"default-dockercfg-nbw99\"" Oct 30 21:42:50.892801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.892772 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"distinct-2-w0-691598-kserve-self-signed-certs\"" Oct 30 21:42:50.892801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.892833 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"openshift-service-ca.crt\"" Oct 30 21:42:50.892801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.892856 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-691598\"/\"kube-root-ca.crt\"" Oct 30 21:42:50.896063 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.896039 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 21:42:50.975765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.963922 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.976345 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.981701 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.981826 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.981957 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82q4f\" (UniqueName: \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.981993 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.982043 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.982276 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.986759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.982549 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.992642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.991401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:50.992921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:50.992739 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:51.046035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:51.045966 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82q4f\" (UniqueName: \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\") pod \"distinct-2-w0-691598-kserve-85585b8488-grf8b\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:51.102544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:51.102514 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:42:51.506586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:51.506557 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 21:42:51.507934 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:42:51.507907 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0156c42c_075a_4d86_96d5_758b85e0ea4c.slice/crio-366fd9b3b8b1ff4a30f02666fbfc82cbf1777338831b4e48b89ac99bf2c8cf2a WatchSource:0}: Error finding container 366fd9b3b8b1ff4a30f02666fbfc82cbf1777338831b4e48b89ac99bf2c8cf2a: Status 404 returned error can't find the container with id 366fd9b3b8b1ff4a30f02666fbfc82cbf1777338831b4e48b89ac99bf2c8cf2a Oct 30 21:42:51.522621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:51.522572 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" event={"ID":"0156c42c-075a-4d86-96d5-758b85e0ea4c","Type":"ContainerStarted","Data":"366fd9b3b8b1ff4a30f02666fbfc82cbf1777338831b4e48b89ac99bf2c8cf2a"} Oct 30 21:42:52.529520 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:52.529487 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" event={"ID":"0156c42c-075a-4d86-96d5-758b85e0ea4c","Type":"ContainerStarted","Data":"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669"} Oct 30 21:42:52.660356 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:52.660302 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podStartSLOduration=2.660287774 podStartE2EDuration="2.660287774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:52.65710233 +0000 UTC m=+1197.331148865" watchObservedRunningTime="2026-10-30 21:42:52.660287774 +0000 UTC m=+1197.334334324" Oct 30 21:42:53.103497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:42:53.103468 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:43:01.107085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:01.107042 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:43:01.120956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:01.120927 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:43:01.590826 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:01.590794 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:43:05.396298 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:43:05.396254 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:43:11.880273 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:11.880190 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:43:22.587661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.587593 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:43:22.591902 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.591876 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.690045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.690011 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:43:22.704402 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704373 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.704714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704689 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.704833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704817 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.704893 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704870 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v47g\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.704931 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704922 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.704965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.704956 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.705047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.705030 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.705163 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.705136 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.705274 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.705255 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806167 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806127 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806354 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806224 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806354 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806289 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806389 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806422 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v47g\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806452 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806488 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806535 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806594 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806627 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806674 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.806986 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.806960 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.807102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.807062 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.807254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.807235 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.808782 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.808760 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.809065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.809045 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.868546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.868509 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.868748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.868625 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v47g\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:22.906337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:22.906300 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:23.337378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.337349 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:43:23.338791 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:43:23.338763 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc986014_3bca_4681_bc9d_ddec6fe08e48.slice/crio-f1e5a50c227e39be53c2d8aa8e55237de18a2a888d9dfde2b8a93712845ba04e WatchSource:0}: Error finding container f1e5a50c227e39be53c2d8aa8e55237de18a2a888d9dfde2b8a93712845ba04e: Status 404 returned error can't find the container with id f1e5a50c227e39be53c2d8aa8e55237de18a2a888d9dfde2b8a93712845ba04e Oct 30 21:43:23.341249 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.341217 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:43:23.341331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.341284 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:43:23.341331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.341312 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:43:23.748757 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.748713 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" event={"ID":"dc986014-3bca-4681-bc9d-ddec6fe08e48","Type":"ContainerStarted","Data":"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c"} Oct 30 21:43:23.749209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.748765 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" event={"ID":"dc986014-3bca-4681-bc9d-ddec6fe08e48","Type":"ContainerStarted","Data":"f1e5a50c227e39be53c2d8aa8e55237de18a2a888d9dfde2b8a93712845ba04e"} Oct 30 21:43:23.873417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.873362 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podStartSLOduration=1.8733470730000001 podStartE2EDuration="1.873347073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:23.870830227 +0000 UTC m=+1228.544876761" watchObservedRunningTime="2026-10-30 21:43:23.873347073 +0000 UTC m=+1228.547393608" Oct 30 21:43:23.907195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:23.907146 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:24.908461 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:24.908422 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:24.908879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:24.908486 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:25.906677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:25.906648 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:25.906930 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:25.906712 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:26.906919 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:26.906883 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:26.907352 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:26.906955 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:27.906952 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:27.906912 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:27.907451 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:27.906974 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:28.907024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:28.906994 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:28.907423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:28.907064 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:29.907029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:29.906983 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:43:29.907524 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:29.907065 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 21:43:30.911791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:30.911763 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:30.912262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:30.912208 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:30.913044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:30.913027 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:43:35.397000 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:43:35.396958 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:43:54.365762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.365723 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 21:43:54.369906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.369887 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410189 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpf6\" (UniqueName: \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410238 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410308 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410334 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410467 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.410646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.410528 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.475749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.475708 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1de75a\"/\"distinct-2-w4-1de75a-kserve-self-signed-certs\"" Oct 30 21:43:54.478632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.478597 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 21:43:54.549630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.512386 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpf6\" (UniqueName: \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.551761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.551733 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.552101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.552076 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.552250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.552227 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.552567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.552543 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.561633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.552851 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.562042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.557232 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.562185 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.558258 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.562278 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.558586 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.571774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.571735 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.588714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.588425 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.590992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.590961 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpf6\" (UniqueName: \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\") pod \"distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:54.682492 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:54.682440 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:43:55.166150 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:55.166121 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 21:43:55.167818 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:43:55.167790 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda44e0779_a66c_4beb_8147_63cecb20e3cb.slice/crio-ec17520e297ecf3bad968b70cecea6904b88b14a0ef1f3f2d22899d457b2de97 WatchSource:0}: Error finding container ec17520e297ecf3bad968b70cecea6904b88b14a0ef1f3f2d22899d457b2de97: Status 404 returned error can't find the container with id ec17520e297ecf3bad968b70cecea6904b88b14a0ef1f3f2d22899d457b2de97 Oct 30 21:43:56.028744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:56.028704 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" event={"ID":"a44e0779-a66c-4beb-8147-63cecb20e3cb","Type":"ContainerStarted","Data":"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d"} Oct 30 21:43:56.028744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:56.028746 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" event={"ID":"a44e0779-a66c-4beb-8147-63cecb20e3cb","Type":"ContainerStarted","Data":"ec17520e297ecf3bad968b70cecea6904b88b14a0ef1f3f2d22899d457b2de97"} Oct 30 21:43:56.152490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:56.152424 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podStartSLOduration=2.152408158 podStartE2EDuration="2.152408158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:56.150513001 +0000 UTC m=+1260.824559538" watchObservedRunningTime="2026-10-30 21:43:56.152408158 +0000 UTC m=+1260.826454722" Oct 30 21:43:56.683507 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:43:56.683475 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:44:04.683255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:04.683199 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:44:04.697864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:04.697831 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:44:05.091082 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:05.091005 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:44:05.397950 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:44:05.397913 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:44:14.702214 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:14.702176 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:44:16.511509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.511474 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:44:16.515528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.515511 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590425 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590494 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590528 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590571 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590598 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj5t\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590740 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.590896 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.591080 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.591646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.591113 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.619806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.619772 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:44:16.708854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.708810 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.709144 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.709116 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj5t\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.709329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.709309 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.709551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.709533 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.709825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.709805 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.709951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.709933 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.710107 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.710088 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.710245 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.710229 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.710364 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.710347 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.715003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.714968 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.715370 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.715345 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.715729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.715706 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.716658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.716635 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.718647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.716668 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.718647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.717464 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.749647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.742227 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.823058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.822984 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.825960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.825930 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj5t\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\") pod \"e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:16.829072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:16.829019 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:17.376748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:17.376720 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:44:17.411373 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:44:17.411343 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e981faa_c724_46a7_bc34_0882165d274f.slice/crio-1634ea737db6dac1194410bf89d2461ac4679c236b12702948924352288ce839 WatchSource:0}: Error finding container 1634ea737db6dac1194410bf89d2461ac4679c236b12702948924352288ce839: Status 404 returned error can't find the container with id 1634ea737db6dac1194410bf89d2461ac4679c236b12702948924352288ce839 Oct 30 21:44:17.413852 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:17.413819 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:17.413927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:17.413886 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:17.413927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:17.413917 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:18.170057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.170013 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" event={"ID":"9e981faa-c724-46a7-bc34-0882165d274f","Type":"ContainerStarted","Data":"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70"} Oct 30 21:44:18.170533 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.170078 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" event={"ID":"9e981faa-c724-46a7-bc34-0882165d274f","Type":"ContainerStarted","Data":"1634ea737db6dac1194410bf89d2461ac4679c236b12702948924352288ce839"} Oct 30 21:44:18.404554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.404497 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podStartSLOduration=2.404483196 podStartE2EDuration="2.404483196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:18.402312168 +0000 UTC m=+1283.076358705" watchObservedRunningTime="2026-10-30 21:44:18.404483196 +0000 UTC m=+1283.078529731" Oct 30 21:44:18.830051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.830012 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:18.845631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.837772 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:18.845631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:18.837846 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:19.852633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:19.852047 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:19.852633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:19.852110 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:20.849637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:20.849307 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:20.849637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:20.849365 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:21.855160 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:21.855121 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:21.855935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:21.855902 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:22.834650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:22.834061 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:22.834650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:22.834140 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:23.830543 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:23.830508 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:44:23.830969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:23.830562 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 21:44:24.834724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:24.834697 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:24.835180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:24.835167 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:24.835839 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:24.835818 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:44:35.397418 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:44:35.397382 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:44:51.628746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.628710 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:44:51.632987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.632965 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.678337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.678302 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:44:51.682409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.682389 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.714547 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.714514 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.714711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.714585 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.714788 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.714723 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.714788 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.714775 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.714905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.714882 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.715094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.715076 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.715142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.715126 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.715229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.715213 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.715287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.715244 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f7t\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.732380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.732325 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:44:51.781309 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.781277 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:44:51.816247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816218 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816278 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816338 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816360 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816516 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816568 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816673 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816698 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mw6\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.816810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816671 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816731 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816811 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816833 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f7t\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.816916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816893 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816930 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.817088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816971 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.816995 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817039 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.817088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817084 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817123 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.817325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817169 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817233 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817334 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.817661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.817643 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.819389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.819368 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.819556 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.819540 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.884678 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.880895 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.889154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.889091 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f7t\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.918442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918398 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918473 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918496 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918559 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918687 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918741 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mw6\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918839 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918900 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918982 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918930 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.918982 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918966 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.919081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.918985 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.919081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.919043 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.919407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.919383 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.919629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.919585 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.921170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.921152 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.921170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.921163 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.947650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.947627 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:51.979968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.979939 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.980122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.980062 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mw6\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\") pod \"e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:51.995808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:51.995775 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:52.382508 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.382476 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:44:52.383455 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:44:52.383324 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb555670_db7e_432f_b0b3_5dd8150a09f8.slice/crio-590be0f206d50a503a3815cb96cb0cf67268785a7588c48c8d8e4dddb123ccc7 WatchSource:0}: Error finding container 590be0f206d50a503a3815cb96cb0cf67268785a7588c48c8d8e4dddb123ccc7: Status 404 returned error can't find the container with id 590be0f206d50a503a3815cb96cb0cf67268785a7588c48c8d8e4dddb123ccc7 Oct 30 21:44:52.386186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.386140 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:52.386311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.386238 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:52.386311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.386287 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:52.411902 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.411871 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" event={"ID":"fb555670-db7e-432f-b0b3-5dd8150a09f8","Type":"ContainerStarted","Data":"590be0f206d50a503a3815cb96cb0cf67268785a7588c48c8d8e4dddb123ccc7"} Oct 30 21:44:52.433426 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.433403 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:44:52.435808 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:44:52.435776 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5fd30d_e8e0_42d4_950c_9ba9fba264ab.slice/crio-7a199a872fd88d82819ceb37f83d45c790ff194f5282a1ab40f26c964ebfc550 WatchSource:0}: Error finding container 7a199a872fd88d82819ceb37f83d45c790ff194f5282a1ab40f26c964ebfc550: Status 404 returned error can't find the container with id 7a199a872fd88d82819ceb37f83d45c790ff194f5282a1ab40f26c964ebfc550 Oct 30 21:44:52.443307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.443276 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:52.443400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.443348 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:52.443400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:52.443378 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:44:53.420328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.420291 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" event={"ID":"fb555670-db7e-432f-b0b3-5dd8150a09f8","Type":"ContainerStarted","Data":"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02"} Oct 30 21:44:53.422181 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.422148 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" event={"ID":"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab","Type":"ContainerStarted","Data":"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335"} Oct 30 21:44:53.422326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.422203 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" event={"ID":"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab","Type":"ContainerStarted","Data":"7a199a872fd88d82819ceb37f83d45c790ff194f5282a1ab40f26c964ebfc550"} Oct 30 21:44:53.656634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.651068 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podStartSLOduration=2.651045326 podStartE2EDuration="2.651045326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:53.594815858 +0000 UTC m=+1318.268862395" watchObservedRunningTime="2026-10-30 21:44:53.651045326 +0000 UTC m=+1318.325091866" Oct 30 21:44:53.728564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.728465 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podStartSLOduration=2.728449194 podStartE2EDuration="2.728449194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:53.725790555 +0000 UTC m=+1318.399837091" watchObservedRunningTime="2026-10-30 21:44:53.728449194 +0000 UTC m=+1318.402495730" Oct 30 21:44:53.951635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.948959 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:53.980302 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.977260 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.980302 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.977321 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:53.996092 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.996056 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:44:53.997857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.997835 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.997939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:53.997883 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:54.950632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:54.949967 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.950632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:54.950029 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:54.997136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:54.997099 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.997312 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:54.997150 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:55.951633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:55.950951 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:55.951633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:55.951012 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:55.997035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:55.997002 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:55.997202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:55.997075 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:56.950630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:56.950052 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:56.950630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:56.950137 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:56.996162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:56.996127 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:56.996575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:56.996190 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:57.951067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:57.951028 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:57.951807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:57.951763 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:57.996135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:57.996106 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:57.996365 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:57.996345 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:58.948914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:58.948879 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:44:58.949093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:58.948931 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 21:44:58.997010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:58.996976 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:44:58.997398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:58.997035 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 21:44:59.953137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:59.953107 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:59.953505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:59.953484 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:44:59.954553 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:44:59.954531 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:45:00.001484 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.001451 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:45:00.001899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.001763 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:45:00.002601 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.002582 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:45:00.178180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.178145 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:00.183431 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.183411 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.190927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.190897 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:00.195970 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.195944 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.221093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.219746 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwc5r\" (UniqueName: \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.221093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.220780 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnwc\" (UniqueName: \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.221093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.220861 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.221093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.221003 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.237084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.236883 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 21:45:00.238311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.238281 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:00.308347 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.308313 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:00.322450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.322416 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwc5r\" (UniqueName: \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.322658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.322518 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnwc\" (UniqueName: \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.322658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.322553 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.322658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.322579 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.323367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.323347 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.323642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.323589 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.385486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.385451 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnwc\" (UniqueName: \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\") pod \"maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.385830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.385797 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwc5r\" (UniqueName: \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\") pod \"maas-api-key-cleanup-29889945-kcstm\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.509702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.506432 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:00.539633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.531298 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:00.982906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.982875 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:00.983500 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:00.983470 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2d70dc_7e27_48dd_9a52_27a3e0c612dc.slice/crio-191028dd04263c604a0b46391f9f529970331da3ef46ebb51fb9c8739093b203 WatchSource:0}: Error finding container 191028dd04263c604a0b46391f9f529970331da3ef46ebb51fb9c8739093b203: Status 404 returned error can't find the container with id 191028dd04263c604a0b46391f9f529970331da3ef46ebb51fb9c8739093b203 Oct 30 21:45:00.985811 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:00.985789 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:45:01.032077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:01.032054 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:01.033223 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:01.033200 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2378bc68_b2d5_4f58_a552_6f866abfc2a0.slice/crio-067feab8ca843338b4c9a3ebc606ffccc6ffbbc5ca416f9b7da11fef1a60453e WatchSource:0}: Error finding container 067feab8ca843338b4c9a3ebc606ffccc6ffbbc5ca416f9b7da11fef1a60453e: Status 404 returned error can't find the container with id 067feab8ca843338b4c9a3ebc606ffccc6ffbbc5ca416f9b7da11fef1a60453e Oct 30 21:45:01.561527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:01.561491 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerStarted","Data":"067feab8ca843338b4c9a3ebc606ffccc6ffbbc5ca416f9b7da11fef1a60453e"} Oct 30 21:45:01.562880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:01.562854 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerStarted","Data":"191028dd04263c604a0b46391f9f529970331da3ef46ebb51fb9c8739093b203"} Oct 30 21:45:01.906738 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:01.906702 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:45:01.906959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:01.906921 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" containerID="cri-o://691023665c23579fd09af814e1c3e9a0d512b2425449eb5b77fbce5fde325e27" gracePeriod=30 Oct 30 21:45:05.400677 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:05.400644 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:45:05.600006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:05.599904 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerStarted","Data":"4da265fa6af494f03d947c05da2f67f68489d40f30306bd0921460e10970ad37"} Oct 30 21:45:05.715850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:05.715801 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" podStartSLOduration=1.706924215 podStartE2EDuration="5.715786602s" totalImagesPullingTime="4.008862387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.714305809 +0000 UTC m=+1330.388352345" watchObservedRunningTime="2026-10-30 21:45:05.715786602 +0000 UTC m=+1330.389833137" Oct 30 21:45:05.787493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:05.787462 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:45:06.642676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.642173 4525 generic.go:356] "Generic (PLEG): container finished" podID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerID="4da265fa6af494f03d947c05da2f67f68489d40f30306bd0921460e10970ad37" exitCode=60 Oct 30 21:45:06.642676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.642311 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerDied","Data":"4da265fa6af494f03d947c05da2f67f68489d40f30306bd0921460e10970ad37"} Oct 30 21:45:06.643192 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.642814 4525 scope.go:122] "RemoveContainer" containerID="4da265fa6af494f03d947c05da2f67f68489d40f30306bd0921460e10970ad37" Oct 30 21:45:06.668332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.668299 4525 generic.go:356] "Generic (PLEG): container finished" podID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerID="b207a390d7908c74a7e0ca7920ad7049d7da2746bda603510fd7fce9185d0468" exitCode=60 Oct 30 21:45:06.668779 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.668753 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerDied","Data":"b207a390d7908c74a7e0ca7920ad7049d7da2746bda603510fd7fce9185d0468"} Oct 30 21:45:06.669318 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:06.669299 4525 scope.go:122] "RemoveContainer" containerID="b207a390d7908c74a7e0ca7920ad7049d7da2746bda603510fd7fce9185d0468" Oct 30 21:45:07.628655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.578378 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:45:07.628655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.618906 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721694 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721748 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721780 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721898 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721933 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721959 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.721993 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.722024 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pft\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.723808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.722060 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.757688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.734917 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:45:07.823633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.814961 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:45:07.823633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.815122 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891663 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891724 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891754 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891791 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891825 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pft\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.891861 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.892002 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.892040 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.892062 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.892536 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.908038 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.908324 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.924172 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:07.927694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.924447 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.018106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:07.994655 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018587 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018766 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018853 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpkk\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018883 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018905 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.018959 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.019068 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.019804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.019139 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.057486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.028129 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:45:08.088673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.086132 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.112780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.110303 4525 generic.go:356] "Generic (PLEG): container finished" podID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerID="691023665c23579fd09af814e1c3e9a0d512b2425449eb5b77fbce5fde325e27" exitCode=0 Oct 30 21:45:08.112780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.110461 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" event={"ID":"f53b560f-0727-4b17-8ee6-8bd491c102c6","Type":"ContainerDied","Data":"691023665c23579fd09af814e1c3e9a0d512b2425449eb5b77fbce5fde325e27"} Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139205 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139311 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139373 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139471 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139532 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139652 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139714 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpkk\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.140637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.139742 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.153546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.151294 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.153546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.152217 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.153546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.152550 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.153546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.153128 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.163311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.160849 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.163311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.160920 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.163311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.161258 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.167889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.167035 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerStarted","Data":"9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc"} Oct 30 21:45:08.173584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.171934 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerStarted","Data":"d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb"} Oct 30 21:45:08.178666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.178582 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.186224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.186161 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.196905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.196786 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 21:45:08.213180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.213150 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pft\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.228688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.219492 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.304866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.300389 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.304866 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.301811 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.510634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.446094 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.510634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.474293 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:45:08.544638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.515419 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpkk\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.571809 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.571867 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.571901 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.571936 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.571975 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.572065 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.572255 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.572293 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.574205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.572343 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\") pod \"f53b560f-0727-4b17-8ee6-8bd491c102c6\" (UID: \"f53b560f-0727-4b17-8ee6-8bd491c102c6\") " Oct 30 21:45:08.578074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.577080 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:08.578074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.577359 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:08.578074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.577675 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:08.578074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.577920 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:08.578746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.578439 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:08.582275 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.582092 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "kube-api-access-7jh9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:08.582723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.582431 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:08.585381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.585124 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:08.587979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.587565 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy" pod "f53b560f-0727-4b17-8ee6-8bd491c102c6" (UID: "f53b560f-0727-4b17-8ee6-8bd491c102c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700222 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700261 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700280 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700299 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700313 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700327 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53b560f-0727-4b17-8ee6-8bd491c102c6-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700341 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700356 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53b560f-0727-4b17-8ee6-8bd491c102c6-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.703639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.700371 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jh9v\" (UniqueName: \"kubernetes.io/projected/f53b560f-0727-4b17-8ee6-8bd491c102c6-kube-api-access-7jh9v\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:08.780426 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.715202 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\") pod \"e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.780426 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.734860 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:08.780426 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.769126 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:08.835227 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.832831 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:08.874367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:08.874290 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" podStartSLOduration=4.540953394 podStartE2EDuration="8.874247051s" totalImagesPullingTime="4.333293657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:08.700799597 +0000 UTC m=+1333.374846135" watchObservedRunningTime="2026-10-30 21:45:08.874247051 +0000 UTC m=+1333.548293589" Oct 30 21:45:09.152631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.152561 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.168326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.167147 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:45:09.606422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.606320 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.818142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.818100 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:09.828935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.818167 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.927475 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.875923 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:09.927475 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:09.878128 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:45:10.114629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.114393 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.478478 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.478544 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r" event={"ID":"f53b560f-0727-4b17-8ee6-8bd491c102c6","Type":"ContainerDied","Data":"9e55b8c89e28617b49f0fed964895dc886ecacb4e8899af15eb51b88744ca450"} Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.478647 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv"] Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.486355 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.486382 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" Oct 30 21:45:10.501666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.487662 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" containerName="istio-proxy" Oct 30 21:45:10.573947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.573893 4525 scope.go:122] "RemoveContainer" containerID="691023665c23579fd09af814e1c3e9a0d512b2425449eb5b77fbce5fde325e27" Oct 30 21:45:10.643672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.628312 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:45:10.687379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.661594 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.699432 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.707990 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.709807 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.712522 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.712700 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.712791 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.713402 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:10.747626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.714447 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:45:10.858131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.811219 4525 generic.go:356] "Generic (PLEG): container finished" podID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerID="9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc" exitCode=60 Oct 30 21:45:10.858131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.812981 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:45:10.858131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.813021 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerDied","Data":"9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc"} Oct 30 21:45:10.868104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.868070 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:45:10.868369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.868349 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv"] Oct 30 21:45:10.868550 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:10.868534 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:45:11.044710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.044512 4525 scope.go:122] "RemoveContainer" containerID="4da265fa6af494f03d947c05da2f67f68489d40f30306bd0921460e10970ad37" Oct 30 21:45:11.423199 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.423169 4525 scope.go:122] "RemoveContainer" containerID="9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc" Oct 30 21:45:11.423836 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:11.423803 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5_odh-ai-gateway-infra(8d2d70dc-7e27-48dd-9a52-27a3e0c612dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" Oct 30 21:45:11.449638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.441938 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.539632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.526210 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:11.587630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.586412 4525 generic.go:356] "Generic (PLEG): container finished" podID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerID="d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb" exitCode=60 Oct 30 21:45:11.591235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.591212 4525 scope.go:122] "RemoveContainer" containerID="d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb" Oct 30 21:45:11.591997 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:11.591956 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kcstm_odh-ai-gateway-infra(2378bc68-b2d5-4f58-a552-6f866abfc2a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" Oct 30 21:45:11.604710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.595191 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:45:11.604966 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.604933 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerDied","Data":"d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb"} Oct 30 21:45:11.605105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.600773 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.637914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.637889 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:11.710955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.697780 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:11.710955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.697857 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.710955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.698067 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.710955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.701743 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:45:11.748476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.724008 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 21:45:11.748476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.724710 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.748476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.724789 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:45:11.748476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.725131 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:45:11.748476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.734368 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:45:11.765807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.765764 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.792850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.792801 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:11.864800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.863687 4525 scope.go:122] "RemoveContainer" containerID="b207a390d7908c74a7e0ca7920ad7049d7da2746bda603510fd7fce9185d0468" Oct 30 21:45:12.014209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.940268 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:45:12.014209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.959345 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:12.014209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.959459 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:12.014209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:11.959517 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:12.065933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.065896 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5kf\" (UniqueName: \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.070979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.070943 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.071308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.071286 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.178981 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.178937 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.180656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.180548 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.180823 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.179829 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.181097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.181079 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5kf\" (UniqueName: \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.223264 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.223195 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.303768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.282279 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5kf\" (UniqueName: \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\") pod \"maas-api-revoke-sub-d752f06e-rzgsv\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.303768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.282854 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:45:12.385688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.313129 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:45:12.385688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.316384 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:45:12.385688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.325701 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:45:12.385688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.325759 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:45:12.385688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.326104 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kt59r"] Oct 30 21:45:12.542320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.542215 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:12.602212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.602185 4525 scope.go:122] "RemoveContainer" containerID="d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb" Oct 30 21:45:12.602567 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:12.602542 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kcstm_odh-ai-gateway-infra(2378bc68-b2d5-4f58-a552-6f866abfc2a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" Oct 30 21:45:12.604239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.604210 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" event={"ID":"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0","Type":"ContainerStarted","Data":"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b"} Oct 30 21:45:12.604348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.604265 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" event={"ID":"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0","Type":"ContainerStarted","Data":"5c09fde40ad732e2ec4aaff9a0fb6cee8cc0a9fe938864f67de8d98d43ad783a"} Oct 30 21:45:12.609494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.608586 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" event={"ID":"95ba8857-1093-48ab-b352-ce3bcfcb129e","Type":"ContainerStarted","Data":"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa"} Oct 30 21:45:12.609494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.608663 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" event={"ID":"95ba8857-1093-48ab-b352-ce3bcfcb129e","Type":"ContainerStarted","Data":"f668694c81c8cf1a408719435b784026c63a86badc1caafc17e755ab63aea0f7"} Oct 30 21:45:12.613003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.612979 4525 scope.go:122] "RemoveContainer" containerID="9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc" Oct 30 21:45:12.613260 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:12.613233 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5_odh-ai-gateway-infra(8d2d70dc-7e27-48dd-9a52-27a3e0c612dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" Oct 30 21:45:12.863672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.851661 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:12.863672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.851752 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:12.953909 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:12.953883 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv"] Oct 30 21:45:12.957019 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:12.956987 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b47cca_109f_4673_8544_c5ef782641f4.slice/crio-b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33 WatchSource:0}: Error finding container b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33: Status 404 returned error can't find the container with id b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33 Oct 30 21:45:13.474781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.474702 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53b560f-0727-4b17-8ee6-8bd491c102c6" path="/var/lib/kubelet/pods/f53b560f-0727-4b17-8ee6-8bd491c102c6/volumes" Oct 30 21:45:13.622280 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.622246 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" event={"ID":"73b47cca-109f-4673-8544-c5ef782641f4","Type":"ContainerStarted","Data":"c8e8a64819f06bdef4b34ee5cb1e534a44a45e3df3cbf75d314b8d1e14302b34"} Oct 30 21:45:13.622423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.622306 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" event={"ID":"73b47cca-109f-4673-8544-c5ef782641f4","Type":"ContainerStarted","Data":"b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33"} Oct 30 21:45:13.729548 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.729443 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podStartSLOduration=6.729426956 podStartE2EDuration="6.729426956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:13.725988199 +0000 UTC m=+1338.400034731" watchObservedRunningTime="2026-10-30 21:45:13.729426956 +0000 UTC m=+1338.403473491" Oct 30 21:45:13.771708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.764807 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:13.771708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.764875 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:13.788758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.788305 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:13.788758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:13.788366 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:14.153634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.152831 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podStartSLOduration=7.152809828 podStartE2EDuration="7.152809828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:14.15212946 +0000 UTC m=+1338.826175998" watchObservedRunningTime="2026-10-30 21:45:14.152809828 +0000 UTC m=+1338.826856364" Oct 30 21:45:14.421947 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.421888 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" podStartSLOduration=5.421868007 podStartE2EDuration="5.421868007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:14.419588808 +0000 UTC m=+1339.093635347" watchObservedRunningTime="2026-10-30 21:45:14.421868007 +0000 UTC m=+1339.095914547" Oct 30 21:45:14.629203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.629168 4525 generic.go:356] "Generic (PLEG): container finished" podID="73b47cca-109f-4673-8544-c5ef782641f4" containerID="c8e8a64819f06bdef4b34ee5cb1e534a44a45e3df3cbf75d314b8d1e14302b34" exitCode=0 Oct 30 21:45:14.629393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.629252 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" event={"ID":"73b47cca-109f-4673-8544-c5ef782641f4","Type":"ContainerDied","Data":"c8e8a64819f06bdef4b34ee5cb1e534a44a45e3df3cbf75d314b8d1e14302b34"} Oct 30 21:45:14.737142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.737055 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:14.737395 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.737365 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:14.777413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.777374 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:14.781735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:14.781696 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:15.750631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:15.737990 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:15.750631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:15.738046 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:15.789159 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:15.788979 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:15.789159 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:15.789033 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:15.928010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:15.927986 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:16.026349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.026263 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\") pod \"73b47cca-109f-4673-8544-c5ef782641f4\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " Oct 30 21:45:16.026349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.026302 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\") pod \"73b47cca-109f-4673-8544-c5ef782641f4\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " Oct 30 21:45:16.026537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.026415 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\" (UniqueName: \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\") pod \"73b47cca-109f-4673-8544-c5ef782641f4\" (UID: \"73b47cca-109f-4673-8544-c5ef782641f4\") " Oct 30 21:45:16.026756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.026697 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca" pod "73b47cca-109f-4673-8544-c5ef782641f4" (UID: "73b47cca-109f-4673-8544-c5ef782641f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:16.026904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.026850 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-openshift-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.028663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.028637 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca" pod "73b47cca-109f-4673-8544-c5ef782641f4" (UID: "73b47cca-109f-4673-8544-c5ef782641f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:16.028835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.028815 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf" pod "73b47cca-109f-4673-8544-c5ef782641f4" (UID: "73b47cca-109f-4673-8544-c5ef782641f4"). InnerVolumeSpecName "kube-api-access-bk5kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:16.128467 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.128435 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5kf\" (UniqueName: \"kubernetes.io/projected/73b47cca-109f-4673-8544-c5ef782641f4-kube-api-access-bk5kf\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.128467 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.128466 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b47cca-109f-4673-8544-c5ef782641f4-maas-api-xks-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.692541 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.692512 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" Oct 30 21:45:16.692727 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.692557 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv" event={"ID":"73b47cca-109f-4673-8544-c5ef782641f4","Type":"ContainerDied","Data":"b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33"} Oct 30 21:45:16.692727 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.692616 4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2342e8124748e0885079c0365f2b1d758861f24603ddc875e72159de0f98e33" Oct 30 21:45:16.755024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.753043 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:16.755024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.753105 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:16.786031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.785997 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:16.786237 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:16.786044 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:17.740633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:17.737979 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:17.740633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:17.738045 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:17.773631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:17.772933 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:17.773631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:17.772993 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:18.738095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:18.738058 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:18.738413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:18.738368 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:18.771074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:18.771041 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:18.771393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:18.771350 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:19.467046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.466998 4525 scope.go:122] "RemoveContainer" containerID="9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc" Oct 30 21:45:19.467555 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.467165 4525 scope.go:122] "RemoveContainer" containerID="d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb" Oct 30 21:45:19.737101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.737030 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.737101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.737080 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 21:45:19.769939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.769902 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.770087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:19.769968 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 21:45:20.726190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.726158 4525 generic.go:356] "Generic (PLEG): container finished" podID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerID="ed8ba62bbfa2d47446c346714d8f739bba530d3a9aa9d1ed09f32e31e79493c7" exitCode=60 Oct 30 21:45:20.726594 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.726221 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerDied","Data":"ed8ba62bbfa2d47446c346714d8f739bba530d3a9aa9d1ed09f32e31e79493c7"} Oct 30 21:45:20.726594 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.726288 4525 scope.go:122] "RemoveContainer" containerID="d83acaeb5f1ba1c187ca71169e675410eca391d7aacc3fd2d71ac707981964cb" Oct 30 21:45:20.726765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.726752 4525 scope.go:122] "RemoveContainer" containerID="ed8ba62bbfa2d47446c346714d8f739bba530d3a9aa9d1ed09f32e31e79493c7" Oct 30 21:45:20.727121 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:20.727099 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kcstm_odh-ai-gateway-infra(2378bc68-b2d5-4f58-a552-6f866abfc2a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" Oct 30 21:45:20.728399 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.728375 4525 generic.go:356] "Generic (PLEG): container finished" podID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerID="5d638fa993a9e8986fd62fbae272acfb81d6f33e30f259ea4d915327b96cfb1b" exitCode=60 Oct 30 21:45:20.728482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.728413 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerDied","Data":"5d638fa993a9e8986fd62fbae272acfb81d6f33e30f259ea4d915327b96cfb1b"} Oct 30 21:45:20.728794 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.728780 4525 scope.go:122] "RemoveContainer" containerID="5d638fa993a9e8986fd62fbae272acfb81d6f33e30f259ea4d915327b96cfb1b" Oct 30 21:45:20.729075 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:20.729059 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5_odh-ai-gateway-infra(8d2d70dc-7e27-48dd-9a52-27a3e0c612dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" Oct 30 21:45:20.738649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.738624 4525 scope.go:122] "RemoveContainer" containerID="9202fc5a8e63a425237d84f78cf5306e640cc09baea774f4356d3984dd6bafdc" Oct 30 21:45:20.742003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.741986 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:20.742251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.742235 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:20.742936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.742919 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:45:20.774357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.774333 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:20.774678 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.774659 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:20.775538 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:20.775519 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:45:21.852962 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:21.852916 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:22.020829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.020803 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:22.060882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.060851 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\" (UniqueName: \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\") pod \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " Oct 30 21:45:22.061062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.060945 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\") pod \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\" (UID: \"2378bc68-b2d5-4f58-a552-6f866abfc2a0\") " Oct 30 21:45:22.061351 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.061325 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca" pod "2378bc68-b2d5-4f58-a552-6f866abfc2a0" (UID: "2378bc68-b2d5-4f58-a552-6f866abfc2a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.061561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.061547 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2378bc68-b2d5-4f58-a552-6f866abfc2a0-maas-api-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.063211 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.063182 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r" pod "2378bc68-b2d5-4f58-a552-6f866abfc2a0" (UID: "2378bc68-b2d5-4f58-a552-6f866abfc2a0"). InnerVolumeSpecName "kube-api-access-vwc5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.094391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.094364 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:22.163092 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.163060 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwc5r\" (UniqueName: \"kubernetes.io/projected/2378bc68-b2d5-4f58-a552-6f866abfc2a0-kube-api-access-vwc5r\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.304985 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.304958 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:22.365407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.365374 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\" (UniqueName: \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\") pod \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " Oct 30 21:45:22.365572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.365466 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\") pod \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\" (UID: \"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc\") " Oct 30 21:45:22.365851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.365823 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca" pod "8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" (UID: "8d2d70dc-7e27-48dd-9a52-27a3e0c612dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.365993 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.365975 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-maas-api-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.367647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.367621 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc" pod "8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" (UID: "8d2d70dc-7e27-48dd-9a52-27a3e0c612dc"). InnerVolumeSpecName "kube-api-access-7cnwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.467470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.467384 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnwc\" (UniqueName: \"kubernetes.io/projected/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc-kube-api-access-7cnwc\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.746486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.746393 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" Oct 30 21:45:22.746486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.746409 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5" event={"ID":"8d2d70dc-7e27-48dd-9a52-27a3e0c612dc","Type":"ContainerDied","Data":"191028dd04263c604a0b46391f9f529970331da3ef46ebb51fb9c8739093b203"} Oct 30 21:45:22.746486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.746480 4525 scope.go:122] "RemoveContainer" containerID="5d638fa993a9e8986fd62fbae272acfb81d6f33e30f259ea4d915327b96cfb1b" Oct 30 21:45:22.748449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.748416 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" event={"ID":"2378bc68-b2d5-4f58-a552-6f866abfc2a0","Type":"ContainerDied","Data":"067feab8ca843338b4c9a3ebc606ffccc6ffbbc5ca416f9b7da11fef1a60453e"} Oct 30 21:45:22.748696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.748443 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm" Oct 30 21:45:22.756850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.756831 4525 scope.go:122] "RemoveContainer" containerID="ed8ba62bbfa2d47446c346714d8f739bba530d3a9aa9d1ed09f32e31e79493c7" Oct 30 21:45:22.913248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.913213 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:22.919123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:22.919100 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c7d477-29889945-pz7k5"] Oct 30 21:45:23.030924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.030848 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:23.036705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.036678 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kcstm"] Oct 30 21:45:23.475194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.475161 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" path="/var/lib/kubelet/pods/2378bc68-b2d5-4f58-a552-6f866abfc2a0/volumes" Oct 30 21:45:23.475666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.475651 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" path="/var/lib/kubelet/pods/8d2d70dc-7e27-48dd-9a52-27a3e0c612dc/volumes" Oct 30 21:45:23.613785 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.613754 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:45:23.614288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614276 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614289 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614315 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614321 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614338 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614343 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614352 4525 cpu_manager.go:414] "containerMap: removing container" podUID="73b47cca-109f-4673-8544-c5ef782641f4" containerName="revoke-keys" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614356 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b47cca-109f-4673-8544-c5ef782641f4" containerName="revoke-keys" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614366 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614373 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614467 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614476 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614486 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614493 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614500 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:23.614767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.614508 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b47cca-109f-4673-8544-c5ef782641f4" containerName="revoke-keys" Oct 30 21:45:23.619356 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.619337 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685141 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685195 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685300 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685498 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685395 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685549 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685652 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cppq6\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685687 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685856 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.685940 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.685919 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.705753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.705723 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:45:23.787176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787092 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787146 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cppq6\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787165 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787466 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787241 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787466 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787288 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787466 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787341 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787464 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787521 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787667 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787570 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787665 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787681 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787868 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787842 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.787868 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.787858 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.788026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.788009 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.789968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.789938 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.790111 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.790088 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.851636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.851570 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.851762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.851724 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cppq6\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\") pod \"e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:23.940515 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:23.940486 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:24.382850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.382819 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:45:24.383429 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:24.383405 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a81afe_4ebc_47bf_90c5_4039f4646392.slice/crio-ef78dd3eb9dfb19dd5a24150770b126d58a71495b0bc3883ca41f8d10e6a032f WatchSource:0}: Error finding container ef78dd3eb9dfb19dd5a24150770b126d58a71495b0bc3883ca41f8d10e6a032f: Status 404 returned error can't find the container with id ef78dd3eb9dfb19dd5a24150770b126d58a71495b0bc3883ca41f8d10e6a032f Oct 30 21:45:24.386516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.386479 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:24.386591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.386546 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:24.386591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.386576 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:24.772286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.772187 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" event={"ID":"04a81afe-4ebc-47bf-90c5-4039f4646392","Type":"ContainerStarted","Data":"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d"} Oct 30 21:45:24.772286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.772252 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" event={"ID":"04a81afe-4ebc-47bf-90c5-4039f4646392","Type":"ContainerStarted","Data":"ef78dd3eb9dfb19dd5a24150770b126d58a71495b0bc3883ca41f8d10e6a032f"} Oct 30 21:45:24.896114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.893915 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podStartSLOduration=1.893891545 podStartE2EDuration="1.893891545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:24.892829238 +0000 UTC m=+1349.566875776" watchObservedRunningTime="2026-10-30 21:45:24.893891545 +0000 UTC m=+1349.567938081" Oct 30 21:45:24.941576 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:24.941543 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:25.957583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:25.957543 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:25.958116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:25.957628 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:26.953632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:26.953026 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:26.953632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:26.953093 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:27.956065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:27.956026 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:27.956532 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:27.956089 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:28.949634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:28.945173 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:28.949634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:28.945242 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:29.955821 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:29.955784 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:29.957653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:29.956576 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:30.940978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:30.940940 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:45:30.941147 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:30.941001 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 21:45:31.945254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:31.945216 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:31.945668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:31.945650 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:31.946528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:31.946505 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:45:32.169282 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:32.169249 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:45:32.169476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:32.169456 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" containerID="cri-o://a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1" gracePeriod=30 Oct 30 21:45:35.397358 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:35.397311 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:45:35.533344 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:35.533312 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-cl8lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:45:35.533504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:35.533366 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:45:37.293576 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.293547 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:45:37.294188 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.294164 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:37.294188 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.294187 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2d70dc-7e27-48dd-9a52-27a3e0c612dc" containerName="cleanup" Oct 30 21:45:37.294350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.294224 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:37.294350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.294233 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:37.294415 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.294380 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2378bc68-b2d5-4f58-a552-6f866abfc2a0" containerName="cleanup" Oct 30 21:45:37.297577 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.297554 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.403935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.403895 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:45:37.420409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420372 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420478 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420501 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420526 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htg4d\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420581 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420640 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420685 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.420753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420705 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.421153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.420951 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.464309 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.464285 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:45:37.521363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521337 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521404 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521480 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521505 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521563 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521550 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521684 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521721 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521730 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:37.521847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521747 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.521847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521837 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\") pod \"81086fe3-5ba5-4bec-9b22-fd4718d95727\" (UID: \"81086fe3-5ba5-4bec-9b22-fd4718d95727\") " Oct 30 21:45:37.522098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.521937 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:37.522098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522077 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:37.522212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522097 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:37.522212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522106 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.522212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522139 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:37.522212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522146 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.522454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522431 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.522523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522467 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.522523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522484 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.522886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522779 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.522998 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523042 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523051 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523073 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htg4d\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523147 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523200 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.523401 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523331 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.523401 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523348 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.523401 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523377 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.523554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523485 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81086fe3-5ba5-4bec-9b22-fd4718d95727-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.524295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.523908 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.524295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.524098 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.524295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.524248 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.526690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.526198 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:37.526690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.526338 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "kube-api-access-lh2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:37.526690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.526404 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:37.526690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.526650 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.527256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.527231 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token" pod "81086fe3-5ba5-4bec-9b22-fd4718d95727" (UID: "81086fe3-5ba5-4bec-9b22-fd4718d95727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:37.527897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.527878 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.584676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.584623 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.586203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.586172 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htg4d\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.612314 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.612271 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:37.625898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.625851 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.625898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.625887 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.625898 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.625903 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81086fe3-5ba5-4bec-9b22-fd4718d95727-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.626158 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.625918 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2cc\" (UniqueName: \"kubernetes.io/projected/81086fe3-5ba5-4bec-9b22-fd4718d95727-kube-api-access-lh2cc\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:37.868370 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.868334 4525 generic.go:356] "Generic (PLEG): container finished" podID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerID="a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1" exitCode=0 Oct 30 21:45:37.868551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.868389 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" Oct 30 21:45:37.868551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.868411 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" event={"ID":"81086fe3-5ba5-4bec-9b22-fd4718d95727","Type":"ContainerDied","Data":"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1"} Oct 30 21:45:37.868551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.868475 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx" event={"ID":"81086fe3-5ba5-4bec-9b22-fd4718d95727","Type":"ContainerDied","Data":"9aae206c1c1c3cb86346341b19bc58ba1391fae0e2bc4572e3ba98455b92150c"} Oct 30 21:45:37.868551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.868499 4525 scope.go:122] "RemoveContainer" containerID="a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1" Oct 30 21:45:37.886954 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.886930 4525 scope.go:122] "RemoveContainer" containerID="a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1" Oct 30 21:45:37.887310 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:45:37.887279 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1\": container with ID starting with a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1 not found: ID does not exist" containerID="a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1" Oct 30 21:45:37.887441 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:37.887321 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1"} err="failed to get container status \"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1\": rpc error: code = NotFound desc = could not find container \"a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1\": container with ID starting with a76268fe8d70e76a551d672f710b9ad15829c667bdf34a332b3682856d2760a1 not found: ID does not exist" Oct 30 21:45:38.037183 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.037150 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:45:38.088352 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.088328 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-cl8lx"] Oct 30 21:45:38.091440 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:38.091141 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536f6deb_1e4b_4616_af3a_097a6a5acfcc.slice/crio-20e83b05e2a3821be99b98a027c8edb3874435f59cd5cc0d7bf85f021e778fb7 WatchSource:0}: Error finding container 20e83b05e2a3821be99b98a027c8edb3874435f59cd5cc0d7bf85f021e778fb7: Status 404 returned error can't find the container with id 20e83b05e2a3821be99b98a027c8edb3874435f59cd5cc0d7bf85f021e778fb7 Oct 30 21:45:38.093707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.093684 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:45:38.093926 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.093896 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:38.093997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.093973 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:38.094051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.094018 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:38.877366 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.877323 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" event={"ID":"536f6deb-1e4b-4616-af3a-097a6a5acfcc","Type":"ContainerStarted","Data":"044c39e5a7a957fba0cd1437be163c4fb294506b12fffa9e1eb89c1116dbfdc6"} Oct 30 21:45:38.877907 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:38.877386 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" event={"ID":"536f6deb-1e4b-4616-af3a-097a6a5acfcc","Type":"ContainerStarted","Data":"20e83b05e2a3821be99b98a027c8edb3874435f59cd5cc0d7bf85f021e778fb7"} Oct 30 21:45:39.001772 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:39.001716 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podStartSLOduration=2.001700259 podStartE2EDuration="2.001700259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:38.999673537 +0000 UTC m=+1363.673720074" watchObservedRunningTime="2026-10-30 21:45:39.001700259 +0000 UTC m=+1363.675746796" Oct 30 21:45:39.474597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:39.474563 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" path="/var/lib/kubelet/pods/81086fe3-5ba5-4bec-9b22-fd4718d95727/volumes" Oct 30 21:45:39.612487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:39.612457 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:39.613824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:39.613798 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:39.613971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:39.613852 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:40.613496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:40.613465 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:40.613914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:40.613521 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:41.612796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:41.612764 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:41.612976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:41.612812 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:42.613599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:42.613567 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:42.613995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:42.613636 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:43.613434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:43.613400 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:43.613663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:43.613470 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:44.613062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:44.613027 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:44.613220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:44.613096 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:45.613355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:45.613322 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:45:45.613960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:45.613383 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 21:45:47.355629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.300691 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:45:47.393774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.393723 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.420629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.398425 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.699634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.689099 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:47.699634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.689186 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.728634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.725796 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:47.787634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.778883 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:45:47.950630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:47.950487 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.154592 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.154545 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.189635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.183710 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.213695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.213174 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.218738 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.218691 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.340750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.340702 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.341450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.341426 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:45:48.395791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.395697 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.396771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.396747 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:45:48.529715 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.529596 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:48.559854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.559826 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:48.677024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.676990 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:45:48.816686 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.802034 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 21:45:48.816686 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.813647 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:45:48.839935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.839892 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:48.862811 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.842272 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:45:49.015959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.964713 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.015959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:48.965085 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:45:49.201748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.190571 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 21:45:49.201748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.190901 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:45:49.235472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.209655 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:45:49.235472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.221176 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:45:49.235472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.221248 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:45:49.271660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.261861 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.311118 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.311086 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:45:49.344761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.344711 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:45:49.425770 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.425711 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.488689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.425939 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.821114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.803713 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:49.842758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:49.842706 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.004447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.004371 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.192636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.192173 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.354969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.333373 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:45:50.386624 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.386560 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:45:50.405634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.396030 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.496963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.496850 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 21:45:50.540654 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.505968 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.634801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.544317 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:45:50.849720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.811626 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.849720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.841726 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:45:50.849720 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.843824 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:50.867848 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:50.867814 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:45:51.032367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.032315 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.324803 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.324738 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.444632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.419735 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.556634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.552744 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:45:51.634690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.575776 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.634690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.577287 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:45:51.634690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.581904 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:45:51.634690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.597843 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:45:51.691657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.665274 4525 cpu_manager.go:414] "containerMap: removing container" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" Oct 30 21:45:51.691657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.665304 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" Oct 30 21:45:51.691657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.665445 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="81086fe3-5ba5-4bec-9b22-fd4718d95727" containerName="istio-proxy" Oct 30 21:45:51.752737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.752697 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.807631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.791059 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.807970 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808113 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808100 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808295 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808391 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6pn\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808536 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808523 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808716 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.808905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.808885 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.809139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.809125 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.948740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.936261 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 21:45:51.948740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.947218 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d4t8c" podUID="5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:51.975636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.967964 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.994385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.994333 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:51.995391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.995366 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.000635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.995707 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.001042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.001007 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.009946 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.009910 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.013631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.010206 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.018808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.018759 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.046901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.046864 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6pn\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.060656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.046776 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.209876 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.175692 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:51.995299 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.113282 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.113547 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.135423 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1225cc53-openshift-default-dockercfg-974xv\"" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.176947 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.053573 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.210095 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.243672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.210371 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.253973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.253927 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 21:45:52.285568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.285516 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.289411 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.289200 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:45:52.289922 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.289892 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6pn\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\") pod \"e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.366484 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.366453 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:52.379653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.378735 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.454838 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.454782 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.532007 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.527442 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.535063 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.532879 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:45:52.614734 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.550335 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:45:52.614734 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.612486 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:45:52.731766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.729748 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:52.738780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.738752 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:45:52.772472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.756423 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:45:52.858860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.858832 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:45:52.859185 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.859169 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:45:52.891065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:52.889131 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:45:53.358042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.358006 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:45:53.361727 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.361688 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:53.361944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.361911 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:53.362093 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.361976 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:45:53.750097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.750061 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:45:53.754194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.754175 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:53.808402 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.808337 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1d456a94\"" Oct 30 21:45:53.809109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.809084 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1d456a94-dockercfg-7bcc6\"" Oct 30 21:45:53.814270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.814240 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:45:53.859632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.856808 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq82j\" (UniqueName: \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:53.859632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:53.856965 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.015682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.010287 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.015682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.010662 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq82j\" (UniqueName: \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.029273 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.028579 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.100330 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.100287 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:45:54.102106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.102076 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq82j\" (UniqueName: \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\") pod \"payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.126812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.126775 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:45:54.127013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.126958 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.153827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.153783 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqss\" (UniqueName: \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.154024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.153880 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.244488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.244451 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerStarted","Data":"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523"} Oct 30 21:45:54.244693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.244514 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerStarted","Data":"31cfe99cd52d096cf7c71cf5a77bca35934fa4715efe941ce228b11f0330c1dd"} Oct 30 21:45:54.255180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.255152 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqss\" (UniqueName: \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.255340 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.255215 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.256021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.256000 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.317315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.317242 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqss\" (UniqueName: \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\") pod \"payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.369489 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.369461 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:54.379600 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.379573 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:54.386136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.386113 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:54.474034 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.473989 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:54.553640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.550092 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podStartSLOduration=4.5500686 podStartE2EDuration="4.5500686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:54.39774233 +0000 UTC m=+1379.071788868" watchObservedRunningTime="2026-10-30 21:45:54.5500686 +0000 UTC m=+1379.224115137" Oct 30 21:45:54.765652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.765620 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:45:54.766292 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:54.766264 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615a5167_8942_4bd8_a755_7ae4ad937280.slice/crio-6f347b09a0bb4adc65d53378a67b3baad4e9bb4c5928d337e4c275c23b9aa7d2 WatchSource:0}: Error finding container 6f347b09a0bb4adc65d53378a67b3baad4e9bb4c5928d337e4c275c23b9aa7d2: Status 404 returned error can't find the container with id 6f347b09a0bb4adc65d53378a67b3baad4e9bb4c5928d337e4c275c23b9aa7d2 Oct 30 21:45:54.997749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:54.997721 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:45:54.998498 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:45:54.998468 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abf0665_922c_4b67_a733_fa2c43f758b7.slice/crio-eba16fbd54cd73043f24547f2952ccb1f70c5c2943a22b40392a688e036b5441 WatchSource:0}: Error finding container eba16fbd54cd73043f24547f2952ccb1f70c5c2943a22b40392a688e036b5441: Status 404 returned error can't find the container with id eba16fbd54cd73043f24547f2952ccb1f70c5c2943a22b40392a688e036b5441 Oct 30 21:45:55.252307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.252217 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" event={"ID":"7abf0665-922c-4b67-a733-fa2c43f758b7","Type":"ContainerStarted","Data":"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467"} Oct 30 21:45:55.252307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.252280 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:45:55.252307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.252301 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" event={"ID":"7abf0665-922c-4b67-a733-fa2c43f758b7","Type":"ContainerStarted","Data":"eba16fbd54cd73043f24547f2952ccb1f70c5c2943a22b40392a688e036b5441"} Oct 30 21:45:55.254028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.253991 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" event={"ID":"615a5167-8942-4bd8-a755-7ae4ad937280","Type":"ContainerStarted","Data":"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578"} Oct 30 21:45:55.254158 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.254052 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" event={"ID":"615a5167-8942-4bd8-a755-7ae4ad937280","Type":"ContainerStarted","Data":"6f347b09a0bb4adc65d53378a67b3baad4e9bb4c5928d337e4c275c23b9aa7d2"} Oct 30 21:45:55.254211 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.254185 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:55.254260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.254213 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:45:55.254999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.254980 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:45:55.376691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.376642 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" podStartSLOduration=2.376627244 podStartE2EDuration="2.376627244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:55.374231722 +0000 UTC m=+1380.048278260" watchObservedRunningTime="2026-10-30 21:45:55.376627244 +0000 UTC m=+1380.050673781" Oct 30 21:45:55.618894 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:55.618830 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" podStartSLOduration=2.618810926 podStartE2EDuration="2.618810926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:55.616645681 +0000 UTC m=+1380.290692217" watchObservedRunningTime="2026-10-30 21:45:55.618810926 +0000 UTC m=+1380.292857465" Oct 30 21:45:59.728633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.719807 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:45:59.741633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.738537 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.782631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.781950 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:45:59.801109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.790105 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnpj\" (UniqueName: \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.801109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.790366 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.818740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.817903 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e6f16377\"" Oct 30 21:45:59.818740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.818051 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e6f16377-dockercfg-62g94\"" Oct 30 21:45:59.915795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.914628 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.915795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.914777 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnpj\" (UniqueName: \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.922450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.916831 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:45:59.931655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.931570 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:45:59.939849 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.939820 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:45:59.947518 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.947488 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:45:59.976657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:45:59.976072 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnpj\" (UniqueName: \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\") pod \"payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:46:00.016220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.016123 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.016220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.016163 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnfc\" (UniqueName: \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.117992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.117959 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.117992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.117998 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnfc\" (UniqueName: \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.118743 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.118724 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.125542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.125521 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:46:00.180767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.180736 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnfc\" (UniqueName: \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\") pod \"payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.266878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.266786 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:00.505939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.505908 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:46:00.506352 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:00.506331 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8d8e86_96ec_4621_97d9_0e05dc99d013.slice/crio-5b764435c613c8ca5d9e6097aade63bb3b63c59d22f1a8a1190e05cde98a7819 WatchSource:0}: Error finding container 5b764435c613c8ca5d9e6097aade63bb3b63c59d22f1a8a1190e05cde98a7819: Status 404 returned error can't find the container with id 5b764435c613c8ca5d9e6097aade63bb3b63c59d22f1a8a1190e05cde98a7819 Oct 30 21:46:00.645940 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:00.645912 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:46:00.646494 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:00.646471 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04ae709_2300_4642_b609_0802417dc0f3.slice/crio-540eda93b0f463768fa0007725cef4bc03a11f5562ecbec666501adeeb5e6c25 WatchSource:0}: Error finding container 540eda93b0f463768fa0007725cef4bc03a11f5562ecbec666501adeeb5e6c25: Status 404 returned error can't find the container with id 540eda93b0f463768fa0007725cef4bc03a11f5562ecbec666501adeeb5e6c25 Oct 30 21:46:01.359914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.359881 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" event={"ID":"a04ae709-2300-4642-b609-0802417dc0f3","Type":"ContainerStarted","Data":"1e426f6eae2db1cc511a1f1c3c37f06dca82055bfa4b5bfc849526d41e6ac917"} Oct 30 21:46:01.360423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.359937 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" event={"ID":"a04ae709-2300-4642-b609-0802417dc0f3","Type":"ContainerStarted","Data":"540eda93b0f463768fa0007725cef4bc03a11f5562ecbec666501adeeb5e6c25"} Oct 30 21:46:01.360423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.359970 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:01.361851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.361827 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" event={"ID":"cb8d8e86-96ec-4621-97d9-0e05dc99d013","Type":"ContainerStarted","Data":"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a"} Oct 30 21:46:01.361946 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.361866 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" event={"ID":"cb8d8e86-96ec-4621-97d9-0e05dc99d013","Type":"ContainerStarted","Data":"5b764435c613c8ca5d9e6097aade63bb3b63c59d22f1a8a1190e05cde98a7819"} Oct 30 21:46:01.361946 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.361903 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:46:01.480638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.480565 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" podStartSLOduration=2.480536327 podStartE2EDuration="2.480536327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:01.477495678 +0000 UTC m=+1386.151542215" watchObservedRunningTime="2026-10-30 21:46:01.480536327 +0000 UTC m=+1386.154582864" Oct 30 21:46:01.632580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:01.629723 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" podStartSLOduration=2.629699355 podStartE2EDuration="2.629699355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:01.605126518 +0000 UTC m=+1386.279173081" watchObservedRunningTime="2026-10-30 21:46:01.629699355 +0000 UTC m=+1386.303745894" Oct 30 21:46:05.396909 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:05.396868 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:46:06.264449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:06.264362 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:46:06.264651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:06.264454 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:46:07.019062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.019025 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:46:07.023496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.023477 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.070660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.070594 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgsz\" (UniqueName: \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.070852 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.070767 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.077004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.076981 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1225cc53\"" Oct 30 21:46:07.082760 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.082736 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:46:07.277450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.277367 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1225cc53-dockercfg-52kln\"" Oct 30 21:46:07.292664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.292631 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:46:07.308045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.305512 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgsz\" (UniqueName: \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.308378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.308352 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.312633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.309811 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.317841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.317813 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:46:07.318315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.318234 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.373644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.373560 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgsz\" (UniqueName: \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\") pod \"payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.443844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.427166 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7s94\" (UniqueName: \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.443844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.427335 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.461289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.454297 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:46:07.461289 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.458959 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.493781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.493752 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:46:07.499361 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.499330 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:46:07.499528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.499422 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.505099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.505075 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.528899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.528802 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.528911 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsc6\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.528970 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.528995 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529126 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529188 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529240 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529220 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529265 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529339 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7s94\" (UniqueName: \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.529421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529393 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529475 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529461 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k99k\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529532 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529521 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529746 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.529877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529782 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.529931 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529907 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.529986 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.529971 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.530044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.530016 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.530693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.530109 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.530693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.530131 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.530693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.530363 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.530894 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.530780 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.552547 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.552512 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:46:07.595791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.595733 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:46:07.614713 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.614655 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7s94\" (UniqueName: \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\") pod \"payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.631884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.631843 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.631884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.631889 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.632151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.631962 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632014 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsc6\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632056 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.632151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632116 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632155 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.632349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632214 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632266 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632293 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.632349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632319 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.632629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632359 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632363 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.632629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632444 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmbx\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.632629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632531 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.632629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632588 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k99k\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.632879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632660 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.632879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632810 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.632879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.632859 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633140 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633112 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633397 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633455 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633495 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633543 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633579 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633728 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633785 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633807 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.633833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633832 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.634806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633884 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.634806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633893 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.634806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.633921 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.634806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.634054 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.634806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.634424 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.635178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.634867 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.635178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.635132 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.635849 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.635824 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.636055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.636035 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.638221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.638193 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:07.639069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.639048 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.640753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.640721 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.661558 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.659041 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:07.672914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.665596 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:46:07.708321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.704088 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsc6\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.708321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.704283 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\") pod \"e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.708321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.708166 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.708321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.708269 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k99k\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738110 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738165 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738205 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738239 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738361 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738412 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.738473 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.739032 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.739993 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.740170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.740034 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmbx\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.743105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.740261 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.743105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.740786 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.743105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.741039 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.743105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.741449 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.743317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.743111 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.753511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.753484 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.786688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.786586 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:07.803724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.803662 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmbx\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.805702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.805677 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:07.815919 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.815888 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:07.823642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:07.823443 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:08.046337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.046294 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:46:08.047920 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:08.047890 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda45076_9f06_4340_8b89_a896fa01f626.slice/crio-dc478d7343ac2a1082a560bfb2a5f160734e9dc5f7f991844c7cb8dc1643a427 WatchSource:0}: Error finding container dc478d7343ac2a1082a560bfb2a5f160734e9dc5f7f991844c7cb8dc1643a427: Status 404 returned error can't find the container with id dc478d7343ac2a1082a560bfb2a5f160734e9dc5f7f991844c7cb8dc1643a427 Oct 30 21:46:08.096361 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.096335 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:46:08.097147 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:08.097120 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7107559_bdfd_419f_a397_2c11dcc164b6.slice/crio-847a0d3353efb7e3a6e4f0522249b400a05265c0edbf25fd8f2b6617819c4933 WatchSource:0}: Error finding container 847a0d3353efb7e3a6e4f0522249b400a05265c0edbf25fd8f2b6617819c4933: Status 404 returned error can't find the container with id 847a0d3353efb7e3a6e4f0522249b400a05265c0edbf25fd8f2b6617819c4933 Oct 30 21:46:08.200235 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:08.200207 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffcd584_3c93_4ffa_b948_6abeb9be9733.slice/crio-67a84c338819bc8df1982bc67401c1702d42cb69354dc067fa187824f662d3fe WatchSource:0}: Error finding container 67a84c338819bc8df1982bc67401c1702d42cb69354dc067fa187824f662d3fe: Status 404 returned error can't find the container with id 67a84c338819bc8df1982bc67401c1702d42cb69354dc067fa187824f662d3fe Oct 30 21:46:08.201526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.201503 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:46:08.203084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.203048 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.203155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.203125 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.203193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.203166 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.248309 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.248277 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:46:08.249277 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:08.249257 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf309456_53e5_4c32_a46d_4155dcc35dbb.slice/crio-3a6fa2da4a339cce9b08829c1713961e69351cdbd53ce3d49b50b6460841a56d WatchSource:0}: Error finding container 3a6fa2da4a339cce9b08829c1713961e69351cdbd53ce3d49b50b6460841a56d: Status 404 returned error can't find the container with id 3a6fa2da4a339cce9b08829c1713961e69351cdbd53ce3d49b50b6460841a56d Oct 30 21:46:08.252568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.252527 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.252568 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:08.252557 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ce90b0_b056_49bb_8e08_85af6bb2a480.slice/crio-d4aab0b49ac95dd3d121991a4a93e6fe14e849f0a6ceaef7d8b27927776bf945 WatchSource:0}: Error finding container d4aab0b49ac95dd3d121991a4a93e6fe14e849f0a6ceaef7d8b27927776bf945: Status 404 returned error can't find the container with id d4aab0b49ac95dd3d121991a4a93e6fe14e849f0a6ceaef7d8b27927776bf945 Oct 30 21:46:08.252735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.252658 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.252735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.252705 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.254076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.254057 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:46:08.255251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.255219 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.255330 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.255302 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.255392 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.255344 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:08.472899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.472838 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" event={"ID":"32ce90b0-b056-49bb-8e08-85af6bb2a480","Type":"ContainerStarted","Data":"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689"} Oct 30 21:46:08.473107 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.472918 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" event={"ID":"32ce90b0-b056-49bb-8e08-85af6bb2a480","Type":"ContainerStarted","Data":"d4aab0b49ac95dd3d121991a4a93e6fe14e849f0a6ceaef7d8b27927776bf945"} Oct 30 21:46:08.475032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.474778 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" event={"ID":"e7107559-bdfd-419f-a397-2c11dcc164b6","Type":"ContainerStarted","Data":"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f"} Oct 30 21:46:08.475729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.475710 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:08.476028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.476009 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" event={"ID":"e7107559-bdfd-419f-a397-2c11dcc164b6","Type":"ContainerStarted","Data":"847a0d3353efb7e3a6e4f0522249b400a05265c0edbf25fd8f2b6617819c4933"} Oct 30 21:46:08.476437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.476415 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" event={"ID":"cf309456-53e5-4c32-a46d-4155dcc35dbb","Type":"ContainerStarted","Data":"0a8ed2404770fba0ccd1ea2db67398fda888997851746e8f29e5cb2e92544a0e"} Oct 30 21:46:08.476499 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.476461 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" event={"ID":"cf309456-53e5-4c32-a46d-4155dcc35dbb","Type":"ContainerStarted","Data":"3a6fa2da4a339cce9b08829c1713961e69351cdbd53ce3d49b50b6460841a56d"} Oct 30 21:46:08.477930 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.477865 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" event={"ID":"2ffcd584-3c93-4ffa-b948-6abeb9be9733","Type":"ContainerStarted","Data":"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44"} Oct 30 21:46:08.477930 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.477907 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" event={"ID":"2ffcd584-3c93-4ffa-b948-6abeb9be9733","Type":"ContainerStarted","Data":"67a84c338819bc8df1982bc67401c1702d42cb69354dc067fa187824f662d3fe"} Oct 30 21:46:08.480295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.479883 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" event={"ID":"bda45076-9f06-4340-8b89-a896fa01f626","Type":"ContainerStarted","Data":"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3"} Oct 30 21:46:08.480295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.479915 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" event={"ID":"bda45076-9f06-4340-8b89-a896fa01f626","Type":"ContainerStarted","Data":"dc478d7343ac2a1082a560bfb2a5f160734e9dc5f7f991844c7cb8dc1643a427"} Oct 30 21:46:08.480295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.480027 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:08.601075 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.600962 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podStartSLOduration=1.600948102 podStartE2EDuration="1.600948102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:08.597667737 +0000 UTC m=+1393.271714276" watchObservedRunningTime="2026-10-30 21:46:08.600948102 +0000 UTC m=+1393.274994706" Oct 30 21:46:08.718338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.718273 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" podStartSLOduration=2.718252098 podStartE2EDuration="2.718252098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:08.713193488 +0000 UTC m=+1393.387240026" watchObservedRunningTime="2026-10-30 21:46:08.718252098 +0000 UTC m=+1393.392298670" Oct 30 21:46:08.786999 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.786965 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:08.818631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.816972 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:08.827634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.824131 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:08.835635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:08.832070 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" podStartSLOduration=1.83205113 podStartE2EDuration="1.83205113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:08.831358211 +0000 UTC m=+1393.505404758" watchObservedRunningTime="2026-10-30 21:46:08.83205113 +0000 UTC m=+1393.506097667" Oct 30 21:46:09.307046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.304861 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podStartSLOduration=2.304839092 podStartE2EDuration="2.304839092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:08.984421807 +0000 UTC m=+1393.658468346" watchObservedRunningTime="2026-10-30 21:46:09.304839092 +0000 UTC m=+1393.978885629" Oct 30 21:46:09.307046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.305031 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podStartSLOduration=2.305020789 podStartE2EDuration="2.305020789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:09.27751867 +0000 UTC m=+1393.951565210" watchObservedRunningTime="2026-10-30 21:46:09.305020789 +0000 UTC m=+1393.979067326" Oct 30 21:46:09.813659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.800997 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.813659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.801063 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:09.817700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.817669 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.817858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.817730 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:09.825952 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.825914 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.826086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:09.825964 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:10.805635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.791016 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.805635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.791079 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:10.816465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.816435 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.816654 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.816484 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:10.824596 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.824571 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.824763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:10.824649 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:11.803164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.803124 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.803448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.803415 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:11.824208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.824160 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.825864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.825805 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:11.826162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.826125 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.826250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:11.826189 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:12.461632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.460857 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:46:12.617635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.617312 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:46:12.827723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.827355 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.827723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.827412 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:12.852800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.852760 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.884960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.884920 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:12.885233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.873758 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.895806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:12.895744 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:13.817660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.797494 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.817660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.797555 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:13.819623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.819576 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.819872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.819837 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:13.824324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.824296 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:13.824492 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:13.824469 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:14.790006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.789959 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.796997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.796953 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:14.880635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.817537 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.899955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.899911 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:14.900204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.890759 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:14.900326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:14.900306 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:15.787459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.787422 4525 patch_prober.go:29] interesting pod/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.787739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.787489 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 21:46:15.820664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.819028 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.820664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.819087 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:46:15.826194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.826162 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:46:15.826422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:15.826392 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 21:46:16.792746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.792707 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:16.793164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.793143 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:16.793931 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.793916 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:46:16.820593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.820567 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:16.820912 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.820892 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:16.821501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.821486 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:46:16.843435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.843376 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:16.843848 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.843828 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:16.844582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:16.844563 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:46:19.582454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:19.582424 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:46:20.608109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:20.608081 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:46:22.514996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.514965 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:46:22.519871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.519849 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.533821 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.533796 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:46:22.641797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.641761 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.641971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.641825 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.641971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.641897 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642019 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642064 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642240 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642221 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slg2l\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642351 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642323 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642381 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.642490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.642400 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.743578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.743537 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.743578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.743578 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.743871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.743707 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.743871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.743753 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.743979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.743891 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744008 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744030 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744076 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744107 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744134 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744222 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg2l\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744305 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744297 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744571 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744548 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.744695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.744683 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.746266 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.746238 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.746711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.746691 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.805249 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.805175 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.805424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.805401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slg2l\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\") pod \"e2e-worker-w5-c7d477-openshift-default-776595695-966nx\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:22.836656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:22.836600 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:23.271266 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:23.271235 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:46:23.271479 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:23.271453 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7548d7d5_cd89_47d3_bab9_04ea46828673.slice/crio-8bf27fe7aea8241f1466a3dcd87dd04bbd3ddb1373d099088cbebf80a4046575 WatchSource:0}: Error finding container 8bf27fe7aea8241f1466a3dcd87dd04bbd3ddb1373d099088cbebf80a4046575: Status 404 returned error can't find the container with id 8bf27fe7aea8241f1466a3dcd87dd04bbd3ddb1373d099088cbebf80a4046575 Oct 30 21:46:23.274143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:23.274109 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:23.274240 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:23.274174 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:23.274240 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:23.274224 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:24.009519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.009480 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" event={"ID":"7548d7d5-cd89-47d3-bab9-04ea46828673","Type":"ContainerStarted","Data":"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd"} Oct 30 21:46:24.009984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.009553 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" event={"ID":"7548d7d5-cd89-47d3-bab9-04ea46828673","Type":"ContainerStarted","Data":"8bf27fe7aea8241f1466a3dcd87dd04bbd3ddb1373d099088cbebf80a4046575"} Oct 30 21:46:24.230164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.230104 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podStartSLOduration=2.230087399 podStartE2EDuration="2.230087399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:24.179912681 +0000 UTC m=+1408.853959219" watchObservedRunningTime="2026-10-30 21:46:24.230087399 +0000 UTC m=+1408.904133936" Oct 30 21:46:24.850633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.850568 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:24.850879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.850802 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:24.850879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:24.850851 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:25.848643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:25.847628 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:25.848643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:25.847689 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:26.365301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:26.365249 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:46:26.846632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:26.841945 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:26.846632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:26.842011 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:27.840631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:27.840007 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:27.840631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:27.840073 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:28.356762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.356708 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.412751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.412706 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.490760 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.490711 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.536762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.536712 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.640778 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.640732 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.793768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.793712 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.814134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.814083 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:28.840125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.840089 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:28.840495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.840460 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:28.858098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:28.856703 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:46:29.151822 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.151770 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.197768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.197715 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.400825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.400782 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.401640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.401458 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:46:29.423017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.422915 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:46:29.423989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.423964 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:46:29.491815 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.491772 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.503637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.499665 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:46:29.613642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.585803 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:46:29.613642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.586765 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:46:29.769690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.768699 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.800171 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.800125 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.801791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.801761 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:46:29.802134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.802118 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:46:29.817151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.817113 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.872824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.858112 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:29.900638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.875156 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:46:30.448973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.448944 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:46:30.449644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:29.928957 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:30.505629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.471460 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:30.508899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.426999 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.510455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.213723 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:46:30.511808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.415493 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.528626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.528581 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:46:30.529321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.498774 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.610698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.579804 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 21:46:30.610698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.601761 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 21:46:30.849419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.837181 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.849419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.837552 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:30.849419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.841758 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:30.849419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.841968 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:30.890633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:30.881330 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:46:31.331819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.331772 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.516043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.515995 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 21:46:31.612092 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.611991 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.630630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.626750 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.630630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.628834 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.662746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.662705 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.697041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.663038 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.697442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.663063 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.890141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.890094 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.928186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.925900 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:31.928186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.926152 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:31.928186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.926209 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:31.959944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.946803 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 21:46:31.992528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.979072 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 21:46:31.992528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.990741 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 21:46:31.992528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:31.992097 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:32.043519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.043490 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 21:46:32.043818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.043804 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 21:46:32.044058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.044046 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:46:32.054345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.048114 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:46:32.068377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.068040 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:46:32.837391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.837346 4525 patch_prober.go:29] interesting pod/e2e-worker-w5-c7d477-openshift-default-776595695-966nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:46:32.837595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:32.837405 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 21:46:34.719767 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:34.719725 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:34.720256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:34.720231 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:34.721160 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:34.721141 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:46:35.398399 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:35.398366 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:46:35.992414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:35.992377 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:46:35.998247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:35.998222 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.052278 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.052246 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1d456a94-serving-cert\"" Oct 30 21:46:36.059886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.059854 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:46:36.080655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.080587 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.080837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.080755 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.080837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.080818 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sm9r\" (UniqueName: \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.080925 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.080896 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.182154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.182120 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.182338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.182246 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.182408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.182333 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sm9r\" (UniqueName: \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.182408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.182383 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.182514 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:36.182476 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-serving-cert: secret "maas-api-e2e-ait-1d456a94-serving-cert" not found Oct 30 21:46:36.182584 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:36.182570 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls podName:79bbf1f0-fd31-4419-9e37-ab08f99ef32a nodeName:}" failed. No retries permitted until 2026-10-30 21:46:36.682547427 +0000 UTC m=+1421.356593948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls") pod "maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" (UID: "79bbf1f0-fd31-4419-9e37-ab08f99ef32a") : secret "maas-api-e2e-ait-1d456a94-serving-cert" not found Oct 30 21:46:36.183206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.183179 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.185010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.184990 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.296016 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.295933 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sm9r\" (UniqueName: \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.708723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.708693 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.711307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.711279 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") pod \"maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:36.915752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:36.915719 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:37.295888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:37.295861 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:46:37.297072 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:37.297040 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bbf1f0_fd31_4419_9e37_ab08f99ef32a.slice/crio-6d2d7ef315be5970d758c04527f02b1534dcfa2ebfa10038e04cd8e8327c4ff8 WatchSource:0}: Error finding container 6d2d7ef315be5970d758c04527f02b1534dcfa2ebfa10038e04cd8e8327c4ff8: Status 404 returned error can't find the container with id 6d2d7ef315be5970d758c04527f02b1534dcfa2ebfa10038e04cd8e8327c4ff8 Oct 30 21:46:37.757663 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:37.757623 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" event={"ID":"79bbf1f0-fd31-4419-9e37-ab08f99ef32a","Type":"ContainerStarted","Data":"6d2d7ef315be5970d758c04527f02b1534dcfa2ebfa10038e04cd8e8327c4ff8"} Oct 30 21:46:38.771495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:38.771452 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" event={"ID":"79bbf1f0-fd31-4419-9e37-ab08f99ef32a","Type":"ContainerStarted","Data":"bde145f8b5611f15c838c54a17013bdf971d5e667464c2375849f6d4a811fdc5"} Oct 30 21:46:38.771957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:38.771530 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:38.898762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:38.898698 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" podStartSLOduration=3.098426101 podStartE2EDuration="3.898672196s" totalImagesPullingTime="800.246095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:38.890054226 +0000 UTC m=+1423.564100763" watchObservedRunningTime="2026-10-30 21:46:38.898672196 +0000 UTC m=+1423.572718732" Oct 30 21:46:44.783994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:44.783961 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:46:46.011746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.011706 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:46:46.016241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.016210 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.077024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.076997 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:46:46.098109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.098066 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e6f16377-serving-cert\"" Oct 30 21:46:46.198630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.198580 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85zl\" (UniqueName: \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.198802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.198728 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.198921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.198904 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.199003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.198990 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.303087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.302987 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85zl\" (UniqueName: \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.304005 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.303983 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.304513 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.304486 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.305571 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.305553 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.306794 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.305417 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.306973 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:46.304256 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-serving-cert: secret "maas-api-e2e-ait-e6f16377-serving-cert" not found Oct 30 21:46:46.307151 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:46.307139 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls podName:7f55a064-3bd9-4475-9cf9-4866ac57ab83 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:46.807117974 +0000 UTC m=+1431.481164498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls") pod "maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" (UID: "7f55a064-3bd9-4475-9cf9-4866ac57ab83") : secret "maas-api-e2e-ait-e6f16377-serving-cert" not found Oct 30 21:46:46.344806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.344774 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.398662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.397330 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85zl\" (UniqueName: \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.815707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.815671 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.818177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.818157 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") pod \"maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:46.931557 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:46.931524 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:47.315844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.315817 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:46:47.316313 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:47.316267 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f55a064_3bd9_4475_9cf9_4866ac57ab83.slice/crio-1d8eda5652ae554337f270917c21d0746a507e62fa35206f45cc02862e8accb7 WatchSource:0}: Error finding container 1d8eda5652ae554337f270917c21d0746a507e62fa35206f45cc02862e8accb7: Status 404 returned error can't find the container with id 1d8eda5652ae554337f270917c21d0746a507e62fa35206f45cc02862e8accb7 Oct 30 21:46:47.803900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.803870 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:46:47.809357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.809334 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.829992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.829959 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830004 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830088 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830141 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830194 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830217 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830312 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830329 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfl5n\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.830516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.830422 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.835236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.835206 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" event={"ID":"7f55a064-3bd9-4475-9cf9-4866ac57ab83","Type":"ContainerStarted","Data":"1d8eda5652ae554337f270917c21d0746a507e62fa35206f45cc02862e8accb7"} Oct 30 21:46:47.863155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.863113 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lfxgk\"" Oct 30 21:46:47.918656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.918053 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:46:47.932178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932136 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932200 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932240 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932267 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932323 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932354 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932429 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932453 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfl5n\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.932739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.932500 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.933200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.933153 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.933435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.933402 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.934265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.934234 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.935329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.935309 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.935821 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.935796 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.938512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.937804 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.939037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.938971 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.997806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.997776 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:47.998109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:47.997847 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfl5n\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\") pod \"maas-default-gateway-openshift-default-75d7c785f-48np4\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:48.124796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.124767 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:48.565028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.564998 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:46:48.565407 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:48.565108 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c00a89_b44d_4daf_a51e_7ae5e3d09a70.slice/crio-3f8c3263cd6b99f9e411184dabb6e9f0c995519f6cab5c3b1f9edf6337475f3e WatchSource:0}: Error finding container 3f8c3263cd6b99f9e411184dabb6e9f0c995519f6cab5c3b1f9edf6337475f3e: Status 404 returned error can't find the container with id 3f8c3263cd6b99f9e411184dabb6e9f0c995519f6cab5c3b1f9edf6337475f3e Oct 30 21:46:48.567862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.567829 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:48.567964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.567895 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:48.567964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.567923 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:48.845011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.844909 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" event={"ID":"7f55a064-3bd9-4475-9cf9-4866ac57ab83","Type":"ContainerStarted","Data":"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0"} Oct 30 21:46:48.845195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.845129 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:48.846825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.846800 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" event={"ID":"49c00a89-b44d-4daf-a51e-7ae5e3d09a70","Type":"ContainerStarted","Data":"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53"} Oct 30 21:46:48.846957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.846850 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" event={"ID":"49c00a89-b44d-4daf-a51e-7ae5e3d09a70","Type":"ContainerStarted","Data":"3f8c3263cd6b99f9e411184dabb6e9f0c995519f6cab5c3b1f9edf6337475f3e"} Oct 30 21:46:48.966725 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:48.966666 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" podStartSLOduration=3.177805667 podStartE2EDuration="3.966652641s" totalImagesPullingTime="788.846974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:48.963576589 +0000 UTC m=+1433.637623148" watchObservedRunningTime="2026-10-30 21:46:48.966652641 +0000 UTC m=+1433.640699178" Oct 30 21:46:49.094631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:49.093843 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podStartSLOduration=2.09381994 podStartE2EDuration="2.09381994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.086976932 +0000 UTC m=+1433.761023492" watchObservedRunningTime="2026-10-30 21:46:49.09381994 +0000 UTC m=+1433.767866477" Oct 30 21:46:49.128002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:49.127522 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:49.128002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:49.127678 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:49.128002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:49.127737 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:50.125827 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:50.125791 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:50.126220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:50.125851 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:51.125828 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.125789 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:51.126210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.125844 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:51.259085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.259036 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:46:51.263873 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.263850 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.304060 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.304023 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.304304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.304280 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.304304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.304310 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.304776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.304713 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p92r\" (UniqueName: \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.308439 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.308414 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:46:51.316260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.316238 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1225cc53-serving-cert\"" Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.406292 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p92r\" (UniqueName: \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.406374 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.406417 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.406446 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:51.406684 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-serving-cert: secret "maas-api-e2e-ait-1225cc53-serving-cert" not found Oct 30 21:46:51.407644 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:46:51.406750 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls podName:acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd nodeName:}" failed. No retries permitted until 2026-10-30 21:46:51.906727374 +0000 UTC m=+1436.580773892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls") pod "maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" (UID: "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd") : secret "maas-api-e2e-ait-1225cc53-serving-cert" not found Oct 30 21:46:51.408580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.408461 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.414437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.414407 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.470346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.470312 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p92r\" (UniqueName: \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.957702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.952221 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:51.958359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:51.958336 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") pod \"maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:52.125238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:52.125203 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:52.125409 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:52.125255 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:52.177536 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:52.177483 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:52.590083 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:46:52.590055 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc2aa5e_2cc9_42ef_b0cf_4c5bd9848cfd.slice/crio-0ed0fe9e5bca3c3d9627247349bc8a2430ecd37f0397942d0fd78af29bacfd96 WatchSource:0}: Error finding container 0ed0fe9e5bca3c3d9627247349bc8a2430ecd37f0397942d0fd78af29bacfd96: Status 404 returned error can't find the container with id 0ed0fe9e5bca3c3d9627247349bc8a2430ecd37f0397942d0fd78af29bacfd96 Oct 30 21:46:52.590877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:52.590849 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:46:52.921570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:52.921526 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" event={"ID":"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd","Type":"ContainerStarted","Data":"0ed0fe9e5bca3c3d9627247349bc8a2430ecd37f0397942d0fd78af29bacfd96"} Oct 30 21:46:53.125869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:53.125836 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:53.126045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:53.125891 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:53.956373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:53.956340 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" event={"ID":"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd","Type":"ContainerStarted","Data":"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2"} Oct 30 21:46:53.956910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:53.956480 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:46:54.077463 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:54.077408 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" podStartSLOduration=2.156183209 podStartE2EDuration="3.0773925s" totalImagesPullingTime="921.209291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.075016369 +0000 UTC m=+1438.749062918" watchObservedRunningTime="2026-10-30 21:46:54.0773925 +0000 UTC m=+1438.751439035" Oct 30 21:46:54.125783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:54.125749 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:54.125951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:54.125806 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:54.864875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:54.864841 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:46:55.125858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:55.125824 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:55.126357 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:55.125896 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:56.125431 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:56.125390 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 21:46:56.125646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:56.125458 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 21:46:58.076494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:58.076456 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:58.077097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:58.077055 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:58.078399 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:58.078381 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:46:59.968914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:46:59.968885 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:47:02.938677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.938642 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:47:02.944308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.944287 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986218 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986324 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986353 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986380 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986556 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986645 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tncj\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986688 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986756 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:02.986818 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:02.986784 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.056459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.056422 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:47:03.087694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.087663 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.087924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.087715 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tncj\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.087924 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.087746 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.087961 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088020 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088207 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088211 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088360 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088339 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088378 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088481 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088424 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088785 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088742 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088825 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.088985 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.088936 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.089862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.089840 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.093152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.093107 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.093329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.093305 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.149545 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.149507 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.149733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.149601 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tncj\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\") pod \"maas-default-gateway-openshift-default-75d7c785f-d88rq\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.259371 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.259280 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:03.692168 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:47:03.692135 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb804e39b_fd87_411b_801a_c273b1cdeb45.slice/crio-399c8da17c9d38260a05d40c333edd459066f4e655e2a8a33f8128a6555fa265 WatchSource:0}: Error finding container 399c8da17c9d38260a05d40c333edd459066f4e655e2a8a33f8128a6555fa265: Status 404 returned error can't find the container with id 399c8da17c9d38260a05d40c333edd459066f4e655e2a8a33f8128a6555fa265 Oct 30 21:47:03.692277 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.692209 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:47:03.694935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.694894 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:03.695028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.694978 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:03.695028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:03.695020 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:04.101495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.101408 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" event={"ID":"b804e39b-fd87-411b-801a-c273b1cdeb45","Type":"ContainerStarted","Data":"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f"} Oct 30 21:47:04.101495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.101477 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" event={"ID":"b804e39b-fd87-411b-801a-c273b1cdeb45","Type":"ContainerStarted","Data":"399c8da17c9d38260a05d40c333edd459066f4e655e2a8a33f8128a6555fa265"} Oct 30 21:47:04.224650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.224575 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podStartSLOduration=2.224557927 podStartE2EDuration="2.224557927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:04.221973685 +0000 UTC m=+1448.896020222" watchObservedRunningTime="2026-10-30 21:47:04.224557927 +0000 UTC m=+1448.898604463" Oct 30 21:47:04.260659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.259961 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:04.261967 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.261940 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:04.262091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:04.261990 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:05.260230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:05.260192 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:05.260700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:05.260261 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:05.398090 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:47:05.398055 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:47:06.260209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:06.260174 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:06.260401 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:06.260229 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:07.260054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:07.260020 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:07.260284 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:07.260085 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:08.260338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:08.260304 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:08.260912 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:08.260378 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:09.260114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:09.260080 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:09.260313 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:09.260139 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:10.260298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:10.260260 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:10.260871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:10.260319 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:11.260084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:11.260051 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-d88rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:47:11.260272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:11.260115 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 21:47:12.420509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:12.420477 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:12.421575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:12.421548 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:12.422307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:12.422291 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:47:22.533416 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:22.533378 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:47:34.414911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.414876 4525 generic.go:356] "Generic (PLEG): container finished" podID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerID="ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523" exitCode=0 Oct 30 21:47:34.415403 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.414948 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerDied","Data":"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523"} Oct 30 21:47:34.415403 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.415370 4525 scope.go:122] "RemoveContainer" containerID="ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523" Oct 30 21:47:34.416978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.416952 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:34.417042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.417020 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:34.417079 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:34.417054 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:47:35.402915 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:47:35.402877 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:47:35.423762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:35.423727 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerStarted","Data":"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a"} Oct 30 21:47:36.379755 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:36.379659 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:47:36.381104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:36.381077 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:47:36.381168 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:36.381142 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:47:37.378950 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:37.378913 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:47:37.379455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:37.378963 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:47:38.379218 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:38.379176 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:47:38.379695 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:38.379236 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:47:39.379193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:39.379159 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:47:39.379370 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:39.379223 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:47:40.254775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:40.254737 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:47:40.379157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:40.379124 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:47:40.379338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:40.379188 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 21:47:41.383798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:41.383765 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:47:41.385027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:41.385005 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:47:54.976221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:54.976190 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:47:55.088347 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:55.088314 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:47:59.976881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:47:59.976838 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" podUID="7abf0665-922c-4b67-a733-fa2c43f758b7" containerName="payload-processing" containerID="cri-o://084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467" gracePeriod=25 Oct 30 21:48:00.089696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.089500 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" podUID="615a5167-8942-4bd8-a755-7ae4ad937280" containerName="payload-pre-processing" containerID="cri-o://0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578" gracePeriod=25 Oct 30 21:48:00.276493 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.276106 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:48:00.369329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.369296 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\" (UniqueName: \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\") pod \"7abf0665-922c-4b67-a733-fa2c43f758b7\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " Oct 30 21:48:00.369531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.369418 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\") pod \"7abf0665-922c-4b67-a733-fa2c43f758b7\" (UID: \"7abf0665-922c-4b67-a733-fa2c43f758b7\") " Oct 30 21:48:00.369884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.369854 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume" pod "7abf0665-922c-4b67-a733-fa2c43f758b7" (UID: "7abf0665-922c-4b67-a733-fa2c43f758b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:00.370280 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.370261 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abf0665-922c-4b67-a733-fa2c43f758b7-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:00.371785 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.371759 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss" pod "7abf0665-922c-4b67-a733-fa2c43f758b7" (UID: "7abf0665-922c-4b67-a733-fa2c43f758b7"). InnerVolumeSpecName "kube-api-access-xnqss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:00.394143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.394118 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:48:00.471362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.471327 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\" (UniqueName: \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\") pod \"615a5167-8942-4bd8-a755-7ae4ad937280\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " Oct 30 21:48:00.471666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.471651 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\") pod \"615a5167-8942-4bd8-a755-7ae4ad937280\" (UID: \"615a5167-8942-4bd8-a755-7ae4ad937280\") " Oct 30 21:48:00.472022 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.471994 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume" pod "615a5167-8942-4bd8-a755-7ae4ad937280" (UID: "615a5167-8942-4bd8-a755-7ae4ad937280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:00.472283 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.472266 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615a5167-8942-4bd8-a755-7ae4ad937280-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:00.472343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.472292 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqss\" (UniqueName: \"kubernetes.io/projected/7abf0665-922c-4b67-a733-fa2c43f758b7-kube-api-access-xnqss\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:00.473639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.473592 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j" pod "615a5167-8942-4bd8-a755-7ae4ad937280" (UID: "615a5167-8942-4bd8-a755-7ae4ad937280"). InnerVolumeSpecName "kube-api-access-dq82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:00.574062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.573966 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq82j\" (UniqueName: \"kubernetes.io/projected/615a5167-8942-4bd8-a755-7ae4ad937280-kube-api-access-dq82j\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:00.745678 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.745647 4525 generic.go:356] "Generic (PLEG): container finished" podID="615a5167-8942-4bd8-a755-7ae4ad937280" containerID="0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578" exitCode=0 Oct 30 21:48:00.745853 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.745695 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" Oct 30 21:48:00.745853 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.745719 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" event={"ID":"615a5167-8942-4bd8-a755-7ae4ad937280","Type":"ContainerDied","Data":"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578"} Oct 30 21:48:00.745853 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.745807 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc" event={"ID":"615a5167-8942-4bd8-a755-7ae4ad937280","Type":"ContainerDied","Data":"6f347b09a0bb4adc65d53378a67b3baad4e9bb4c5928d337e4c275c23b9aa7d2"} Oct 30 21:48:00.745853 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.745835 4525 scope.go:122] "RemoveContainer" containerID="0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578" Oct 30 21:48:00.747512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.747489 4525 generic.go:356] "Generic (PLEG): container finished" podID="7abf0665-922c-4b67-a733-fa2c43f758b7" containerID="084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467" exitCode=0 Oct 30 21:48:00.747687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.747553 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" event={"ID":"7abf0665-922c-4b67-a733-fa2c43f758b7","Type":"ContainerDied","Data":"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467"} Oct 30 21:48:00.747687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.747559 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" Oct 30 21:48:00.747687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.747595 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x" event={"ID":"7abf0665-922c-4b67-a733-fa2c43f758b7","Type":"ContainerDied","Data":"eba16fbd54cd73043f24547f2952ccb1f70c5c2943a22b40392a688e036b5441"} Oct 30 21:48:00.756809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.756727 4525 scope.go:122] "RemoveContainer" containerID="0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578" Oct 30 21:48:00.757017 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:00.756987 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578\": container with ID starting with 0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578 not found: ID does not exist" containerID="0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578" Oct 30 21:48:00.757120 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.757029 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578"} err="failed to get container status \"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578\": rpc error: code = NotFound desc = could not find container \"0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578\": container with ID starting with 0b2264264947e67b509ab3f006a7dc5a4e6cdd460f9ca4b5e112e01f350e1578 not found: ID does not exist" Oct 30 21:48:00.757120 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.757046 4525 scope.go:122] "RemoveContainer" containerID="084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467" Oct 30 21:48:00.767178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.767160 4525 scope.go:122] "RemoveContainer" containerID="084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467" Oct 30 21:48:00.767474 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:00.767447 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467\": container with ID starting with 084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467 not found: ID does not exist" containerID="084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467" Oct 30 21:48:00.767522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.767486 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467"} err="failed to get container status \"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467\": rpc error: code = NotFound desc = could not find container \"084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467\": container with ID starting with 084d612f6bd628d0364aaa9c9c9104cd8307b6181dd618966b36390374aa1467 not found: ID does not exist" Oct 30 21:48:00.915858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.915824 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:48:00.919592 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:00.919572 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d456a94-554c9dccdc-28rdc"] Oct 30 21:48:01.029730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:01.029700 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:48:01.035634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:01.035585 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d456a94-7cd478598-v5h2x"] Oct 30 21:48:01.474642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:01.474590 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615a5167-8942-4bd8-a755-7ae4ad937280" path="/var/lib/kubelet/pods/615a5167-8942-4bd8-a755-7ae4ad937280/volumes" Oct 30 21:48:01.475071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:01.475058 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abf0665-922c-4b67-a733-fa2c43f758b7" path="/var/lib/kubelet/pods/7abf0665-922c-4b67-a733-fa2c43f758b7/volumes" Oct 30 21:48:05.397744 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:05.397707 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:48:05.897862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:05.897823 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:48:06.009627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:06.009574 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:48:08.446073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:08.446037 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:48:08.571670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:08.571638 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:48:10.898472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:10.898417 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" podUID="e7107559-bdfd-419f-a397-2c11dcc164b6" containerName="payload-processing" containerID="cri-o://3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f" gracePeriod=25 Oct 30 21:48:11.011038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.010987 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" podUID="bda45076-9f06-4340-8b89-a896fa01f626" containerName="payload-pre-processing" containerID="cri-o://c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3" gracePeriod=25 Oct 30 21:48:11.193957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.193925 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:48:11.289396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.289359 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\" (UniqueName: \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\") pod \"e7107559-bdfd-419f-a397-2c11dcc164b6\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " Oct 30 21:48:11.289589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.289462 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\") pod \"e7107559-bdfd-419f-a397-2c11dcc164b6\" (UID: \"e7107559-bdfd-419f-a397-2c11dcc164b6\") " Oct 30 21:48:11.289861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.289835 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume" pod "e7107559-bdfd-419f-a397-2c11dcc164b6" (UID: "e7107559-bdfd-419f-a397-2c11dcc164b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:11.290387 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.290365 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7107559-bdfd-419f-a397-2c11dcc164b6-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:11.291657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.291632 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94" pod "e7107559-bdfd-419f-a397-2c11dcc164b6" (UID: "e7107559-bdfd-419f-a397-2c11dcc164b6"). InnerVolumeSpecName "kube-api-access-s7s94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:11.316665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.316633 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:48:11.391664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.391627 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\") pod \"bda45076-9f06-4340-8b89-a896fa01f626\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " Oct 30 21:48:11.391851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.391759 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\" (UniqueName: \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\") pod \"bda45076-9f06-4340-8b89-a896fa01f626\" (UID: \"bda45076-9f06-4340-8b89-a896fa01f626\") " Oct 30 21:48:11.392079 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.392054 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume" pod "bda45076-9f06-4340-8b89-a896fa01f626" (UID: "bda45076-9f06-4340-8b89-a896fa01f626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:11.392411 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.392391 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda45076-9f06-4340-8b89-a896fa01f626-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:11.392634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.392416 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7s94\" (UniqueName: \"kubernetes.io/projected/e7107559-bdfd-419f-a397-2c11dcc164b6-kube-api-access-s7s94\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:11.394499 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.394471 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz" pod "bda45076-9f06-4340-8b89-a896fa01f626" (UID: "bda45076-9f06-4340-8b89-a896fa01f626"). InnerVolumeSpecName "kube-api-access-ppgsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:11.719685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.716257 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppgsz\" (UniqueName: \"kubernetes.io/projected/bda45076-9f06-4340-8b89-a896fa01f626-kube-api-access-ppgsz\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:11.873128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.873089 4525 generic.go:356] "Generic (PLEG): container finished" podID="bda45076-9f06-4340-8b89-a896fa01f626" containerID="c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3" exitCode=0 Oct 30 21:48:11.873348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.873139 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" Oct 30 21:48:11.873348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.873174 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" event={"ID":"bda45076-9f06-4340-8b89-a896fa01f626","Type":"ContainerDied","Data":"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3"} Oct 30 21:48:11.873348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.873232 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl" event={"ID":"bda45076-9f06-4340-8b89-a896fa01f626","Type":"ContainerDied","Data":"dc478d7343ac2a1082a560bfb2a5f160734e9dc5f7f991844c7cb8dc1643a427"} Oct 30 21:48:11.873348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.873262 4525 scope.go:122] "RemoveContainer" containerID="c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3" Oct 30 21:48:11.875000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.874976 4525 generic.go:356] "Generic (PLEG): container finished" podID="e7107559-bdfd-419f-a397-2c11dcc164b6" containerID="3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f" exitCode=0 Oct 30 21:48:11.875106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.875088 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" Oct 30 21:48:11.875186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.875091 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" event={"ID":"e7107559-bdfd-419f-a397-2c11dcc164b6","Type":"ContainerDied","Data":"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f"} Oct 30 21:48:11.875244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.875212 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n" event={"ID":"e7107559-bdfd-419f-a397-2c11dcc164b6","Type":"ContainerDied","Data":"847a0d3353efb7e3a6e4f0522249b400a05265c0edbf25fd8f2b6617819c4933"} Oct 30 21:48:11.886050 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.886023 4525 scope.go:122] "RemoveContainer" containerID="c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3" Oct 30 21:48:11.886435 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:11.886401 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3\": container with ID starting with c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3 not found: ID does not exist" containerID="c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3" Oct 30 21:48:11.886521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.886445 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3"} err="failed to get container status \"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3\": rpc error: code = NotFound desc = could not find container \"c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3\": container with ID starting with c2c472ab3cd9030075e4a00d2f59c0a558e62da126c69f7713e3ee3c998acfb3 not found: ID does not exist" Oct 30 21:48:11.886521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.886472 4525 scope.go:122] "RemoveContainer" containerID="3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f" Oct 30 21:48:11.899053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.899026 4525 scope.go:122] "RemoveContainer" containerID="3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f" Oct 30 21:48:11.899976 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:11.899946 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f\": container with ID starting with 3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f not found: ID does not exist" containerID="3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f" Oct 30 21:48:11.900088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:11.899987 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f"} err="failed to get container status \"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f\": rpc error: code = NotFound desc = could not find container \"3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f\": container with ID starting with 3cab9c851eb4e329c6c03a6f9c6f51cfac71bf3bcaff920e18b732ea8361b60f not found: ID does not exist" Oct 30 21:48:12.041864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:12.041780 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:48:12.091051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:12.091010 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1225cc53-78c49d6df9-pqvvl"] Oct 30 21:48:12.159548 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:12.159504 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:48:12.163544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:12.163517 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1225cc53-75d8c9688b-vcw2n"] Oct 30 21:48:13.446745 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.446677 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" podUID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" containerName="payload-processing" containerID="cri-o://12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a" gracePeriod=25 Oct 30 21:48:13.479430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.479393 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda45076-9f06-4340-8b89-a896fa01f626" path="/var/lib/kubelet/pods/bda45076-9f06-4340-8b89-a896fa01f626/volumes" Oct 30 21:48:13.480125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.480101 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7107559-bdfd-419f-a397-2c11dcc164b6" path="/var/lib/kubelet/pods/e7107559-bdfd-419f-a397-2c11dcc164b6/volumes" Oct 30 21:48:13.572248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.572123 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" podUID="a04ae709-2300-4642-b609-0802417dc0f3" containerName="payload-pre-processing" containerID="cri-o://1e426f6eae2db1cc511a1f1c3c37f06dca82055bfa4b5bfc849526d41e6ac917" gracePeriod=25 Oct 30 21:48:13.933203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.932806 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:48:13.970090 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.970006 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\" (UniqueName: \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\") pod \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " Oct 30 21:48:13.970450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.970427 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\") pod \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\" (UID: \"cb8d8e86-96ec-4621-97d9-0e05dc99d013\") " Oct 30 21:48:13.972072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.970924 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume" pod "cb8d8e86-96ec-4621-97d9-0e05dc99d013" (UID: "cb8d8e86-96ec-4621-97d9-0e05dc99d013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:13.978391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.978189 4525 generic.go:356] "Generic (PLEG): container finished" podID="a04ae709-2300-4642-b609-0802417dc0f3" containerID="1e426f6eae2db1cc511a1f1c3c37f06dca82055bfa4b5bfc849526d41e6ac917" exitCode=0 Oct 30 21:48:13.978832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.978215 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" event={"ID":"a04ae709-2300-4642-b609-0802417dc0f3","Type":"ContainerDied","Data":"1e426f6eae2db1cc511a1f1c3c37f06dca82055bfa4b5bfc849526d41e6ac917"} Oct 30 21:48:13.999096 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.999060 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" event={"ID":"cb8d8e86-96ec-4621-97d9-0e05dc99d013","Type":"ContainerDied","Data":"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a"} Oct 30 21:48:13.999398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.999377 4525 scope.go:122] "RemoveContainer" containerID="12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a" Oct 30 21:48:13.999660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.989287 4525 generic.go:356] "Generic (PLEG): container finished" podID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" containerID="12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a" exitCode=0 Oct 30 21:48:14.017637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:13.989357 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" Oct 30 21:48:14.018186 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.004110 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj" pod "cb8d8e86-96ec-4621-97d9-0e05dc99d013" (UID: "cb8d8e86-96ec-4621-97d9-0e05dc99d013"). InnerVolumeSpecName "kube-api-access-ncnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:14.018382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.018246 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d" event={"ID":"cb8d8e86-96ec-4621-97d9-0e05dc99d013","Type":"ContainerDied","Data":"5b764435c613c8ca5d9e6097aade63bb3b63c59d22f1a8a1190e05cde98a7819"} Oct 30 21:48:14.100944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.097934 4525 scope.go:122] "RemoveContainer" containerID="12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a" Oct 30 21:48:14.104887 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.104473 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnpj\" (UniqueName: \"kubernetes.io/projected/cb8d8e86-96ec-4621-97d9-0e05dc99d013-kube-api-access-ncnpj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.104887 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.104505 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8d8e86-96ec-4621-97d9-0e05dc99d013-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.105352 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:14.105234 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a\": container with ID starting with 12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a not found: ID does not exist" containerID="12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a" Oct 30 21:48:14.105352 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.105279 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a"} err="failed to get container status \"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a\": rpc error: code = NotFound desc = could not find container \"12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a\": container with ID starting with 12cc9cd65c62548fd959e724de3b6264a4262d7363f7cdcebd2ff2a7252a5c3a not found: ID does not exist" Oct 30 21:48:14.148719 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.148677 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-48np4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:48:14.149290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.148981 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:14.214781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.214754 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:48:14.307377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.307299 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\") pod \"a04ae709-2300-4642-b609-0802417dc0f3\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " Oct 30 21:48:14.307377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.307340 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\" (UniqueName: \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\") pod \"a04ae709-2300-4642-b609-0802417dc0f3\" (UID: \"a04ae709-2300-4642-b609-0802417dc0f3\") " Oct 30 21:48:14.307703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.307678 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume" pod "a04ae709-2300-4642-b609-0802417dc0f3" (UID: "a04ae709-2300-4642-b609-0802417dc0f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:14.307830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.307818 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a04ae709-2300-4642-b609-0802417dc0f3-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.309555 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.309536 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc" pod "a04ae709-2300-4642-b609-0802417dc0f3" (UID: "a04ae709-2300-4642-b609-0802417dc0f3"). InnerVolumeSpecName "kube-api-access-xlnfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:14.409404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.409372 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlnfc\" (UniqueName: \"kubernetes.io/projected/a04ae709-2300-4642-b609-0802417dc0f3-kube-api-access-xlnfc\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.464682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.464582 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:48:14.472855 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:14.472829 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6f16377-f9844fdd4-lhx8d"] Oct 30 21:48:15.026311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.026282 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" Oct 30 21:48:15.026494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.026320 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7" event={"ID":"a04ae709-2300-4642-b609-0802417dc0f3","Type":"ContainerDied","Data":"540eda93b0f463768fa0007725cef4bc03a11f5562ecbec666501adeeb5e6c25"} Oct 30 21:48:15.026494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.026382 4525 scope.go:122] "RemoveContainer" containerID="1e426f6eae2db1cc511a1f1c3c37f06dca82055bfa4b5bfc849526d41e6ac917" Oct 30 21:48:15.194707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.194672 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:48:15.255029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.254989 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6f16377-6cc45d9755-52kt7"] Oct 30 21:48:15.475311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.475274 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04ae709-2300-4642-b609-0802417dc0f3" path="/var/lib/kubelet/pods/a04ae709-2300-4642-b609-0802417dc0f3/volumes" Oct 30 21:48:15.475801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:15.475785 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" path="/var/lib/kubelet/pods/cb8d8e86-96ec-4621-97d9-0e05dc99d013/volumes" Oct 30 21:48:27.697945 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:27.697913 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:48:27.698390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:27.698107 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" podUID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" containerName="maas-api" containerID="cri-o://bde145f8b5611f15c838c54a17013bdf971d5e667464c2375849f6d4a811fdc5" gracePeriod=30 Oct 30 21:48:28.281564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.279911 4525 generic.go:356] "Generic (PLEG): container finished" podID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" containerID="bde145f8b5611f15c838c54a17013bdf971d5e667464c2375849f6d4a811fdc5" exitCode=0 Oct 30 21:48:28.281564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.280124 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" event={"ID":"79bbf1f0-fd31-4419-9e37-ab08f99ef32a","Type":"ContainerDied","Data":"bde145f8b5611f15c838c54a17013bdf971d5e667464c2375849f6d4a811fdc5"} Oct 30 21:48:28.298628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.297434 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:48:28.383885 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.383833 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") pod \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " Oct 30 21:48:28.384065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.383971 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\") pod \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " Oct 30 21:48:28.384152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.384072 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\") pod \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " Oct 30 21:48:28.384206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.384155 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\" (UniqueName: \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\") pod \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\" (UID: \"79bbf1f0-fd31-4419-9e37-ab08f99ef32a\") " Oct 30 21:48:28.386973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.386933 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs" pod "79bbf1f0-fd31-4419-9e37-ab08f99ef32a" (UID: "79bbf1f0-fd31-4419-9e37-ab08f99ef32a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:28.388080 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.388050 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r" pod "79bbf1f0-fd31-4419-9e37-ab08f99ef32a" (UID: "79bbf1f0-fd31-4419-9e37-ab08f99ef32a"). InnerVolumeSpecName "kube-api-access-5sm9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:28.388718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.388366 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls" pod "79bbf1f0-fd31-4419-9e37-ab08f99ef32a" (UID: "79bbf1f0-fd31-4419-9e37-ab08f99ef32a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:28.388718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.388683 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca" pod "79bbf1f0-fd31-4419-9e37-ab08f99ef32a" (UID: "79bbf1f0-fd31-4419-9e37-ab08f99ef32a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:28.486239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.486158 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:28.486239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.486186 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:28.486239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.486197 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:28.486239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:28.486207 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sm9r\" (UniqueName: \"kubernetes.io/projected/79bbf1f0-fd31-4419-9e37-ab08f99ef32a-kube-api-access-5sm9r\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:29.293964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:29.293921 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" event={"ID":"79bbf1f0-fd31-4419-9e37-ab08f99ef32a","Type":"ContainerDied","Data":"6d2d7ef315be5970d758c04527f02b1534dcfa2ebfa10038e04cd8e8327c4ff8"} Oct 30 21:48:29.294623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:29.294582 4525 scope.go:122] "RemoveContainer" containerID="bde145f8b5611f15c838c54a17013bdf971d5e667464c2375849f6d4a811fdc5" Oct 30 21:48:29.294969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:29.294947 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx" Oct 30 21:48:29.578744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:29.573888 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:48:29.586874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:29.586794 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1d456a94-67bc56fddc-h87cx"] Oct 30 21:48:31.475579 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:31.475548 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" path="/var/lib/kubelet/pods/79bbf1f0-fd31-4419-9e37-ab08f99ef32a/volumes" Oct 30 21:48:34.244683 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.244646 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:48:34.245084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.244900 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" podUID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" containerName="maas-api" containerID="cri-o://90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0" gracePeriod=30 Oct 30 21:48:34.731633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.724792 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:48:34.763630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.760341 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\") pod \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " Oct 30 21:48:34.771362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.771310 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs" pod "7f55a064-3bd9-4475-9cf9-4866ac57ab83" (UID: "7f55a064-3bd9-4475-9cf9-4866ac57ab83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:34.874845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.865400 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\" (UniqueName: \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\") pod \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " Oct 30 21:48:34.874845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.865721 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") pod \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " Oct 30 21:48:34.874845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.865821 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\") pod \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\" (UID: \"7f55a064-3bd9-4475-9cf9-4866ac57ab83\") " Oct 30 21:48:34.874845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.866624 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.936859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.882973 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca" pod "7f55a064-3bd9-4475-9cf9-4866ac57ab83" (UID: "7f55a064-3bd9-4475-9cf9-4866ac57ab83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:34.936859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.903172 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl" pod "7f55a064-3bd9-4475-9cf9-4866ac57ab83" (UID: "7f55a064-3bd9-4475-9cf9-4866ac57ab83"). InnerVolumeSpecName "kube-api-access-b85zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:34.944155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.943436 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls" pod "7f55a064-3bd9-4475-9cf9-4866ac57ab83" (UID: "7f55a064-3bd9-4475-9cf9-4866ac57ab83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:34.968807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.968772 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f55a064-3bd9-4475-9cf9-4866ac57ab83-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.968807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.968811 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f55a064-3bd9-4475-9cf9-4866ac57ab83-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:34.969011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:34.968827 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85zl\" (UniqueName: \"kubernetes.io/projected/7f55a064-3bd9-4475-9cf9-4866ac57ab83-kube-api-access-b85zl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:35.377636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.375131 4525 generic.go:356] "Generic (PLEG): container finished" podID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" containerID="90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0" exitCode=0 Oct 30 21:48:35.377636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.375278 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" event={"ID":"7f55a064-3bd9-4475-9cf9-4866ac57ab83","Type":"ContainerDied","Data":"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0"} Oct 30 21:48:35.377636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.375331 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" event={"ID":"7f55a064-3bd9-4475-9cf9-4866ac57ab83","Type":"ContainerDied","Data":"1d8eda5652ae554337f270917c21d0746a507e62fa35206f45cc02862e8accb7"} Oct 30 21:48:35.377636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.375365 4525 scope.go:122] "RemoveContainer" containerID="90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0" Oct 30 21:48:35.377636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.375526 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9" Oct 30 21:48:35.500935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.498920 4525 scope.go:122] "RemoveContainer" containerID="90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0" Oct 30 21:48:35.502718 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:35.501375 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0\": container with ID starting with 90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0 not found: ID does not exist" containerID="90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0" Oct 30 21:48:35.502718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.501415 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0"} err="failed to get container status \"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0\": rpc error: code = NotFound desc = could not find container \"90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0\": container with ID starting with 90e1cf5a4328ddf43a6cc42700bf9da0183531f0ff6122e266a0ffb606d51ba0 not found: ID does not exist" Oct 30 21:48:35.504543 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.504517 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:48:35.506335 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.506304 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" podUID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" containerName="maas-api" containerID="cri-o://ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2" gracePeriod=30 Oct 30 21:48:35.529415 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:35.529295 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:48:35.570137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.570106 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:48:35.573753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:35.573726 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6f16377-86bd7dd9db-c9nh9"] Oct 30 21:48:36.187840 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.187813 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:48:36.253744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.253707 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\") pod \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " Oct 30 21:48:36.253921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.253782 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") pod \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " Oct 30 21:48:36.253921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.253877 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\" (UniqueName: \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\") pod \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " Oct 30 21:48:36.254041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.254019 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\") pod \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\" (UID: \"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd\") " Oct 30 21:48:36.254166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.254135 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca" pod "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" (UID: "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:36.254833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.254805 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:36.256139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.256107 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls" pod "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" (UID: "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:36.256139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.256123 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs" pod "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" (UID: "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:36.256265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.256164 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r" pod "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" (UID: "acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd"). InnerVolumeSpecName "kube-api-access-8p92r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:36.398680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.395952 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:36.398680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.395986 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:36.398680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.396002 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p92r\" (UniqueName: \"kubernetes.io/projected/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd-kube-api-access-8p92r\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:36.412250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.412216 4525 generic.go:356] "Generic (PLEG): container finished" podID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" containerID="ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2" exitCode=0 Oct 30 21:48:36.412428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.412259 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" event={"ID":"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd","Type":"ContainerDied","Data":"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2"} Oct 30 21:48:36.412428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.412308 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" Oct 30 21:48:36.412428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.412331 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp" event={"ID":"acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd","Type":"ContainerDied","Data":"0ed0fe9e5bca3c3d9627247349bc8a2430ecd37f0397942d0fd78af29bacfd96"} Oct 30 21:48:36.412428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.412351 4525 scope.go:122] "RemoveContainer" containerID="ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2" Oct 30 21:48:36.423257 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.423058 4525 scope.go:122] "RemoveContainer" containerID="ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2" Oct 30 21:48:36.423413 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:36.423389 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2\": container with ID starting with ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2 not found: ID does not exist" containerID="ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2" Oct 30 21:48:36.423488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.423425 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2"} err="failed to get container status \"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2\": rpc error: code = NotFound desc = could not find container \"ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2\": container with ID starting with ce8099e3e0504ca0814c252de29e820c99d84b28ad293903070b1840b6ca13c2 not found: ID does not exist" Oct 30 21:48:36.650686 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.649791 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:48:36.672641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:36.670927 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1225cc53-769bb6f445-jk4lp"] Oct 30 21:48:37.476783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:37.476745 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" path="/var/lib/kubelet/pods/7f55a064-3bd9-4475-9cf9-4866ac57ab83/volumes" Oct 30 21:48:37.477472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:37.477451 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" path="/var/lib/kubelet/pods/acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd/volumes" Oct 30 21:48:38.011993 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:38.011958 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:48:38.012194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:38.012169 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" containerID="cri-o://8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a" gracePeriod=30 Oct 30 21:48:39.544996 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.544965 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:48:39.545585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545567 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e7107559-bdfd-419f-a397-2c11dcc164b6" containerName="payload-processing" Oct 30 21:48:39.545585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545583 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7107559-bdfd-419f-a397-2c11dcc164b6" containerName="payload-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545598 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7abf0665-922c-4b67-a733-fa2c43f758b7" containerName="payload-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545619 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abf0665-922c-4b67-a733-fa2c43f758b7" containerName="payload-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545633 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a04ae709-2300-4642-b609-0802417dc0f3" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545642 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04ae709-2300-4642-b609-0802417dc0f3" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545660 4525 cpu_manager.go:414] "containerMap: removing container" podUID="615a5167-8942-4bd8-a755-7ae4ad937280" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545667 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615a5167-8942-4bd8-a755-7ae4ad937280" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545685 4525 cpu_manager.go:414] "containerMap: removing container" podUID="bda45076-9f06-4340-8b89-a896fa01f626" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545692 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda45076-9f06-4340-8b89-a896fa01f626" containerName="payload-pre-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545699 4525 cpu_manager.go:414] "containerMap: removing container" podUID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" containerName="payload-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545703 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" containerName="payload-processing" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545728 4525 cpu_manager.go:414] "containerMap: removing container" podUID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" containerName="maas-api" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545732 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" containerName="maas-api" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545741 4525 cpu_manager.go:414] "containerMap: removing container" podUID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" containerName="maas-api" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545745 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" containerName="maas-api" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545756 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" containerName="maas-api" Oct 30 21:48:39.545762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545760 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" containerName="maas-api" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545866 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc2aa5e-2cc9-42ef-b0cf-4c5bd9848cfd" containerName="maas-api" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545876 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f55a064-3bd9-4475-9cf9-4866ac57ab83" containerName="maas-api" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545885 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7107559-bdfd-419f-a397-2c11dcc164b6" containerName="payload-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545891 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abf0665-922c-4b67-a733-fa2c43f758b7" containerName="payload-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545897 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04ae709-2300-4642-b609-0802417dc0f3" containerName="payload-pre-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545903 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="615a5167-8942-4bd8-a755-7ae4ad937280" containerName="payload-pre-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545909 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda45076-9f06-4340-8b89-a896fa01f626" containerName="payload-pre-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545916 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8d8e86-96ec-4621-97d9-0e05dc99d013" containerName="payload-processing" Oct 30 21:48:39.546233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.545930 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bbf1f0-fd31-4419-9e37-ab08f99ef32a" containerName="maas-api" Oct 30 21:48:39.551001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.550950 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.790206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.724109 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-21084e54-gw-openshift-default-dockercfg-56lvl\"" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.790843 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791073 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791127 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791203 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791276 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxhs\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791314 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791344 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791462 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.793766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.791497 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:39.950631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.907929 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.049585 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.049724 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.049840 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.049954 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxhs\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.048831 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.049991 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.050045 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:39.909089 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.050241 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.050318 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.051267 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.067650 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.098673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.068737 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.136859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.132504 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxhs\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.148718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.142177 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.148718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.142337 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.157641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.155677 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.157641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.156820 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\") pod \"e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.165445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.165421 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:40.643090 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.643063 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:48:40.643453 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:40.643237 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b965a1d_b306_4d53_8c21_1912a802742c.slice/crio-2334bda96e94533111df2c4320640363e7fc446300b3aaf0d90f2f359923f4f9 WatchSource:0}: Error finding container 2334bda96e94533111df2c4320640363e7fc446300b3aaf0d90f2f359923f4f9: Status 404 returned error can't find the container with id 2334bda96e94533111df2c4320640363e7fc446300b3aaf0d90f2f359923f4f9 Oct 30 21:48:40.645804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.645767 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:40.645911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.645854 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:40.645911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:40.645881 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:41.386419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:41.386384 4525 patch_prober.go:29] interesting pod/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:41.386596 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:41.386446 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:41.510591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:41.510551 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerStarted","Data":"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0"} Oct 30 21:48:41.510780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:41.510635 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerStarted","Data":"2334bda96e94533111df2c4320640363e7fc446300b3aaf0d90f2f359923f4f9"} Oct 30 21:48:41.634836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:41.634787 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podStartSLOduration=2.634772405 podStartE2EDuration="2.634772405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:41.63280872 +0000 UTC m=+1546.306855256" watchObservedRunningTime="2026-10-30 21:48:41.634772405 +0000 UTC m=+1546.308818992" Oct 30 21:48:42.170383 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:42.170355 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:42.176257 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:42.176225 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:42.520040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:42.519954 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:42.521086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:42.521067 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:48:43.326561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.326463 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:48:43.331631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.331163 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.442267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.442240 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fd496e14-openshift-default-dockercfg-mm7s9\"" Oct 30 21:48:43.442456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.442344 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:48:43.465640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465590 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.465891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465658 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.465891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465796 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.465891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465843 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.465891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465881 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.466229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.465952 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.466229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.466067 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mssbw\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.466229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.466155 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.466229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.466207 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.488110 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.480227 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:48:43.592935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.592782 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.593333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.593308 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.593516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.593500 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.593814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.593776 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.594136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.594104 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mssbw\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.594380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.594364 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.594598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.594579 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.595098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.595078 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.595330 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.595279 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.610762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.609119 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.610762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.610089 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.612217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.611453 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.614294 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.614270 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.615135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.615107 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.657238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.657205 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.679752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.675452 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.696162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696132 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.696431 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696408 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.696583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696567 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.696742 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696726 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.696837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696813 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:43.696970 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.696954 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.697157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.697141 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.697315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.697299 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.697447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.697431 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.697595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.697580 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\") pod \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\" (UID: \"d9dd6333-d0ee-4481-8a6b-d68d97e525e2\") " Oct 30 21:48:43.698229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.698200 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:43.698408 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.698379 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:43.698488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.698387 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:43.699499 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.699476 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.699921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.699898 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.700115 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.700100 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.700275 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.700262 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.700405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.699708 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:43.700505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.700384 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mssbw\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.733119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.733079 4525 generic.go:356] "Generic (PLEG): container finished" podID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerID="8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a" exitCode=0 Oct 30 21:48:43.734702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.734679 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" Oct 30 21:48:43.735471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.734917 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerDied","Data":"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a"} Oct 30 21:48:43.735680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.735663 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks" event={"ID":"d9dd6333-d0ee-4481-8a6b-d68d97e525e2","Type":"ContainerDied","Data":"31cfe99cd52d096cf7c71cf5a77bca35934fa4715efe941ce228b11f0330c1dd"} Oct 30 21:48:43.735829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.735789 4525 scope.go:122] "RemoveContainer" containerID="8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a" Oct 30 21:48:43.802756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.802722 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.833971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.830671 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:43.833971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.830765 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:43.833971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.831544 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\") pod \"e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:43.833971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.831790 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:43.836138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.836107 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn" pod "d9dd6333-d0ee-4481-8a6b-d68d97e525e2" (UID: "d9dd6333-d0ee-4481-8a6b-d68d97e525e2"). InnerVolumeSpecName "kube-api-access-vn6pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:43.843126 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.843076 4525 scope.go:122] "RemoveContainer" containerID="ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523" Oct 30 21:48:43.856162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.856137 4525 scope.go:122] "RemoveContainer" containerID="8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a" Oct 30 21:48:43.856591 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:43.856561 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a\": container with ID starting with 8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a not found: ID does not exist" containerID="8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a" Oct 30 21:48:43.856718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.856620 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a"} err="failed to get container status \"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a\": rpc error: code = NotFound desc = could not find container \"8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a\": container with ID starting with 8eebfe5b6cc2f6d13439f8afc790b91b7bb3b10212acc29725305be74724438a not found: ID does not exist" Oct 30 21:48:43.856718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.856647 4525 scope.go:122] "RemoveContainer" containerID="ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523" Oct 30 21:48:43.856990 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:48:43.856962 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523\": container with ID starting with ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523 not found: ID does not exist" containerID="ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523" Oct 30 21:48:43.857108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.856995 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523"} err="failed to get container status \"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523\": rpc error: code = NotFound desc = could not find container \"ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523\": container with ID starting with ce8c9db8f3868068aa5250ac3a7b4462545f446b70aef38ae2ebe8508df13523 not found: ID does not exist" Oct 30 21:48:43.904319 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.904291 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.904319 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.904318 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.904509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.904327 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6pn\" (UniqueName: \"kubernetes.io/projected/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-kube-api-access-vn6pn\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.904509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.904336 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9dd6333-d0ee-4481-8a6b-d68d97e525e2-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:43.947196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:43.947164 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:44.163443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.163409 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:48:44.168568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.168543 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1225cc53-openshift-default-bb8498c97-d8xks"] Oct 30 21:48:44.381413 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:44.381384 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d54bb1_ebb4_4a28_895f_67c22eb20b71.slice/crio-74820522c11a78349ed50f7483998225f0cb981d838acf46ce769d49656e37d2 WatchSource:0}: Error finding container 74820522c11a78349ed50f7483998225f0cb981d838acf46ce769d49656e37d2: Status 404 returned error can't find the container with id 74820522c11a78349ed50f7483998225f0cb981d838acf46ce769d49656e37d2 Oct 30 21:48:44.381764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.381650 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:48:44.384414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.384371 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:44.384514 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.384464 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:44.384554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.384512 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:44.746134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.746037 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerStarted","Data":"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa"} Oct 30 21:48:44.746134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.746102 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerStarted","Data":"74820522c11a78349ed50f7483998225f0cb981d838acf46ce769d49656e37d2"} Oct 30 21:48:44.874512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.874462 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podStartSLOduration=1.874446447 podStartE2EDuration="1.874446447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:44.871797255 +0000 UTC m=+1549.545843790" watchObservedRunningTime="2026-10-30 21:48:44.874446447 +0000 UTC m=+1549.548492982" Oct 30 21:48:44.947307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:44.947274 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:45.476108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:45.476075 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" path="/var/lib/kubelet/pods/d9dd6333-d0ee-4481-8a6b-d68d97e525e2/volumes" Oct 30 21:48:45.953629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:45.953586 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:46.771411 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:46.771383 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:46.772208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:46.772192 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:48:47.277365 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.277327 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:48:47.278750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.278718 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.278975 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.278957 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.279116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.279099 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.279218 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.279208 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.280541 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.279524 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.288872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.288841 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.358003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.357008 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-21084e54-dockercfg-ngddh\"" Oct 30 21:48:47.358003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.357184 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-21084e54\"" Oct 30 21:48:47.405705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.373795 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:48:47.514993 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.460751 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dnj\" (UniqueName: \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.524668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.516948 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.624654 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.619379 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.624654 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.619584 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dnj\" (UniqueName: \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.640674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.640108 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.699311 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:48:47.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.700892 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dd6333-d0ee-4481-8a6b-d68d97e525e2" containerName="istio-proxy" Oct 30 21:48:47.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.706869 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:48:47.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.706975 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.709291 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dnj\" (UniqueName: \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\") pod \"payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.823955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.823914 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kg7p\" (UniqueName: \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.824422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.824147 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.906748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.906653 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:47.925671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.925591 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.926052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.926023 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kg7p\" (UniqueName: \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.926346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.926324 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:47.990744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:47.990680 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kg7p\" (UniqueName: \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\") pod \"payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:48.039757 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.039704 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:48.288709 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.288676 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:48:48.289422 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:48.289395 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea29bba8_9004_4a64_8502_5cce1cbe9eac.slice/crio-81b72613ee3093fafb2a8d38a0afdeafbd7d790c97a4f8988b3b65ce2b12f391 WatchSource:0}: Error finding container 81b72613ee3093fafb2a8d38a0afdeafbd7d790c97a4f8988b3b65ce2b12f391: Status 404 returned error can't find the container with id 81b72613ee3093fafb2a8d38a0afdeafbd7d790c97a4f8988b3b65ce2b12f391 Oct 30 21:48:48.420141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.420114 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:48:48.420926 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:48.420898 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23bc062_9f29_4bc4_9dd9_69a889c372e4.slice/crio-fa359938f011528f3a7f5e802717cb04c821e903d3ecca807ff88061a3acdb8b WatchSource:0}: Error finding container fa359938f011528f3a7f5e802717cb04c821e903d3ecca807ff88061a3acdb8b: Status 404 returned error can't find the container with id fa359938f011528f3a7f5e802717cb04c821e903d3ecca807ff88061a3acdb8b Oct 30 21:48:48.761965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.761931 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:48:48.766461 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.766436 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.788595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.788548 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" event={"ID":"ea29bba8-9004-4a64-8502-5cce1cbe9eac","Type":"ContainerStarted","Data":"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228"} Oct 30 21:48:48.788774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.788658 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:48.788774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.788681 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" event={"ID":"ea29bba8-9004-4a64-8502-5cce1cbe9eac","Type":"ContainerStarted","Data":"81b72613ee3093fafb2a8d38a0afdeafbd7d790c97a4f8988b3b65ce2b12f391"} Oct 30 21:48:48.790298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.790271 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" event={"ID":"a23bc062-9f29-4bc4-9dd9-69a889c372e4","Type":"ContainerStarted","Data":"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80"} Oct 30 21:48:48.790419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.790319 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" event={"ID":"a23bc062-9f29-4bc4-9dd9-69a889c372e4","Type":"ContainerStarted","Data":"fa359938f011528f3a7f5e802717cb04c821e903d3ecca807ff88061a3acdb8b"} Oct 30 21:48:48.790511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.790494 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:48:48.796784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.796762 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:48:48.800897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.800882 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.840754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.840718 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfl44\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.840842 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841001 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841049 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841069 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841120 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841146 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841190 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.841295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.841249 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.874971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.874937 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:48:48.932762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.932727 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:48:48.943920 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.943885 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfl44\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.946033 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.945990 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.946423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.946397 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.946644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.946625 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.946863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.946845 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.947383 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.947364 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9jw\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.947628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.947431 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.947763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.947745 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:48.947931 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.947915 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.954527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.948393 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.955626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.955575 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.976776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.976729 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:48.991715 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.991680 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.012915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.991983 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.099530 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.099584 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.099629 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.099714 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.099750 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.977295 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.966181 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.100544 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.100688 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:48.948988 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.116205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.070359 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.116205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.086587 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfl44\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.116205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.090340 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.170666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.170143 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-89fbn\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.196770 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.196547 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" podStartSLOduration=2.196527059 podStartE2EDuration="2.196527059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:49.045178889 +0000 UTC m=+1553.719225426" watchObservedRunningTime="2026-10-30 21:48:49.196527059 +0000 UTC m=+1553.870573595" Oct 30 21:48:49.202419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.202379 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.202739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.202717 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.202895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.202873 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.203039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.203025 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9jw\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.203230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.203214 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.203687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.203662 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.203957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.203936 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.204117 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.204100 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.204230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.204215 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.204900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.204869 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.205675 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.205653 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.213890 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.207937 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.213890 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.208816 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.213890 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.209517 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.238568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.234728 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.240248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.239259 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.270626 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.270530 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.272039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.272006 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9jw\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\") pod \"maas-default-gateway-openshift-default-75d7c785f-fxrfr\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.320801 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.320739 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" podStartSLOduration=2.320719278 podStartE2EDuration="2.320719278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:49.19813414 +0000 UTC m=+1553.872180678" watchObservedRunningTime="2026-10-30 21:48:49.320719278 +0000 UTC m=+1553.994765816" Oct 30 21:48:49.382656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.382039 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:49.452650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.439174 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:49.895053 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:49.895020 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c035e29_2266_4243_90e7_620553f3087b.slice/crio-19ba9ffe641ee9b7599dcea11b5b4e2be1e1fb40d41bdeb43fa15a36641fbe04 WatchSource:0}: Error finding container 19ba9ffe641ee9b7599dcea11b5b4e2be1e1fb40d41bdeb43fa15a36641fbe04: Status 404 returned error can't find the container with id 19ba9ffe641ee9b7599dcea11b5b4e2be1e1fb40d41bdeb43fa15a36641fbe04 Oct 30 21:48:49.895448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.895427 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:48:49.895699 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:49.895677 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48457e2_c981_400f_8115_29e53b06c69a.slice/crio-863297a4be03d6bae38cdf0e04fbfeb6efba8e62f2f277165ac952ed30bd4790 WatchSource:0}: Error finding container 863297a4be03d6bae38cdf0e04fbfeb6efba8e62f2f277165ac952ed30bd4790: Status 404 returned error can't find the container with id 863297a4be03d6bae38cdf0e04fbfeb6efba8e62f2f277165ac952ed30bd4790 Oct 30 21:48:49.898413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898383 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.898482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898424 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.898482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898447 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.898482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898474 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.898584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898497 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.898584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.898539 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:49.900114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:49.900091 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:48:50.807085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:50.807056 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" event={"ID":"b48457e2-c981-400f-8115-29e53b06c69a","Type":"ContainerStarted","Data":"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9"} Oct 30 21:48:50.807311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:50.807295 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" event={"ID":"b48457e2-c981-400f-8115-29e53b06c69a","Type":"ContainerStarted","Data":"863297a4be03d6bae38cdf0e04fbfeb6efba8e62f2f277165ac952ed30bd4790"} Oct 30 21:48:50.814145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:50.809796 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" event={"ID":"0c035e29-2266-4243-90e7-620553f3087b","Type":"ContainerStarted","Data":"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a"} Oct 30 21:48:50.814145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:50.809904 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" event={"ID":"0c035e29-2266-4243-90e7-620553f3087b","Type":"ContainerStarted","Data":"19ba9ffe641ee9b7599dcea11b5b4e2be1e1fb40d41bdeb43fa15a36641fbe04"} Oct 30 21:48:50.961632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:50.960711 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podStartSLOduration=2.960685147 podStartE2EDuration="2.960685147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:50.928845547 +0000 UTC m=+1555.602892085" watchObservedRunningTime="2026-10-30 21:48:50.960685147 +0000 UTC m=+1555.634731694" Oct 30 21:48:51.067693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.067529 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podStartSLOduration=3.06750649 podStartE2EDuration="3.06750649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.05136797 +0000 UTC m=+1555.725414508" watchObservedRunningTime="2026-10-30 21:48:51.06750649 +0000 UTC m=+1555.741553027" Oct 30 21:48:51.383320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.383273 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:51.384916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.384891 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:51.384994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.384937 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:51.440091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.440056 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:51.459776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.459727 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:51.459973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:51.459794 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:52.395341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.395306 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:52.437636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.411792 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:52.495692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.473765 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:52.586747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.586688 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:52.587369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.587345 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:48:52.891630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.830795 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:48:52.891630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.830918 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.000899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.910512 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fd496e14\"" Oct 30 21:48:53.000899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.910688 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fd496e14-dockercfg-bztwd\"" Oct 30 21:48:53.000899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.914861 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sd8j\" (UniqueName: \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.000899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:52.915022 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.051666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.034344 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:48:53.101633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.089662 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.101633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.090095 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sd8j\" (UniqueName: \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.103167 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.103134 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.127944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.125288 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:48:53.127944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.125408 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.342196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.330032 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86stk\" (UniqueName: \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.343140 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.343102 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.434738 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.434501 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:53.434738 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.434558 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:53.488652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.474055 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86stk\" (UniqueName: \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.488652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.474289 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.488652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.475207 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sd8j\" (UniqueName: \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\") pod \"payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.488652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.475250 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.493317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.493279 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:53.503771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.503731 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:53.577634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.577403 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86stk\" (UniqueName: \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\") pod \"payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:53.725417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.725345 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:53.803836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:53.803804 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:54.222446 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:54.222416 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2012a45a_5810_4c8f_926f_a65be0e22ffc.slice/crio-9196630f55d5008aff5de48c38f3381859644d43a589fadfd6808228769a7c94 WatchSource:0}: Error finding container 9196630f55d5008aff5de48c38f3381859644d43a589fadfd6808228769a7c94: Status 404 returned error can't find the container with id 9196630f55d5008aff5de48c38f3381859644d43a589fadfd6808228769a7c94 Oct 30 21:48:54.224442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.224418 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:48:54.286070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.286006 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:48:54.289121 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:48:54.289053 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab8424a_6792_4bd5_a4c5_79fd8f8e8043.slice/crio-19aa87a9b8c744992b4c8acfa5912bb4ff63820413ed65ed0013250b84629628 WatchSource:0}: Error finding container 19aa87a9b8c744992b4c8acfa5912bb4ff63820413ed65ed0013250b84629628: Status 404 returned error can't find the container with id 19aa87a9b8c744992b4c8acfa5912bb4ff63820413ed65ed0013250b84629628 Oct 30 21:48:54.384265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.383994 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:54.384265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.384053 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:54.450633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.450033 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:54.450633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:54.450115 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:55.005105 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.005068 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" event={"ID":"2012a45a-5810-4c8f-926f-a65be0e22ffc","Type":"ContainerStarted","Data":"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34"} Oct 30 21:48:55.005292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.005130 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" event={"ID":"2012a45a-5810-4c8f-926f-a65be0e22ffc","Type":"ContainerStarted","Data":"9196630f55d5008aff5de48c38f3381859644d43a589fadfd6808228769a7c94"} Oct 30 21:48:55.005292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.005172 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:48:55.007175 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.007155 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" event={"ID":"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043","Type":"ContainerStarted","Data":"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a"} Oct 30 21:48:55.007269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.007189 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" event={"ID":"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043","Type":"ContainerStarted","Data":"19aa87a9b8c744992b4c8acfa5912bb4ff63820413ed65ed0013250b84629628"} Oct 30 21:48:55.007269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.007209 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:48:55.125747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.125685 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" podStartSLOduration=3.125667973 podStartE2EDuration="3.125667973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.122286438 +0000 UTC m=+1559.796332972" watchObservedRunningTime="2026-10-30 21:48:55.125667973 +0000 UTC m=+1559.799714508" Oct 30 21:48:55.265679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.262858 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" podStartSLOduration=3.262836749 podStartE2EDuration="3.262836749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.258940895 +0000 UTC m=+1559.932987432" watchObservedRunningTime="2026-10-30 21:48:55.262836749 +0000 UTC m=+1559.936883286" Oct 30 21:48:55.392810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.387922 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:55.392810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.387986 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:55.440633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.440562 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:55.440820 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:55.440652 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:56.383330 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:56.383278 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:56.383876 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:56.383349 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:56.453632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:56.452770 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:56.453632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:56.452826 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:57.382972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:57.382942 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:57.383152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:57.382993 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:57.439989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:57.439950 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:57.440392 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:57.440017 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:58.383352 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:58.383313 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-89fbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:48:58.383531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:58.383379 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:48:58.440589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:58.440554 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fxrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:48:58.441001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:58.440637 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 21:48:59.128567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.128519 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:48:59.767754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.767723 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:59.768262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.768242 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:59.769132 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.769111 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:48:59.786327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.786295 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:59.786860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.786835 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:59.787948 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.787926 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:48:59.802074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.802047 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:48:59.802248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:48:59.802123 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:49:05.398363 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:49:05.398323 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:49:06.016736 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:06.016710 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:49:06.016944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:06.016790 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:49:25.204044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.204010 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:49:25.209935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.209912 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.256952 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.256917 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.257119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.256962 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.257119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.257024 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw26q\" (UniqueName: \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.257191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.257128 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.267089 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.267058 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:49:25.311554 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.311517 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-21084e54-serving-cert\"" Oct 30 21:49:25.358800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.358755 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.358800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.358802 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.359102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.358840 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw26q\" (UniqueName: \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.359102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.358932 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.360847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.360817 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.361139 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:49:25.361121 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-serving-cert: secret "maas-api-e2e-migrate-21084e54-serving-cert" not found Oct 30 21:49:25.361332 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:49:25.361319 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls podName:9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:25.861297512 +0000 UTC m=+1590.535344041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls") pod "maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" (UID: "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8") : secret "maas-api-e2e-migrate-21084e54-serving-cert" not found Oct 30 21:49:25.362162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.362115 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.420990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.420963 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw26q\" (UniqueName: \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.918019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.917975 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:25.920812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:25.920785 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") pod \"maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:26.125984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:26.125944 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:26.508546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:26.508519 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:49:26.508929 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:49:26.508909 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db1caa2_0e9c_413a_b6e1_1e24c9d4dbf8.slice/crio-492484dc3bd7a25bb8890bcd93d6e7ac22964408105f9ecc9747e6d085d6d54f WatchSource:0}: Error finding container 492484dc3bd7a25bb8890bcd93d6e7ac22964408105f9ecc9747e6d085d6d54f: Status 404 returned error can't find the container with id 492484dc3bd7a25bb8890bcd93d6e7ac22964408105f9ecc9747e6d085d6d54f Oct 30 21:49:27.442445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:27.442408 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" event={"ID":"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8","Type":"ContainerStarted","Data":"492484dc3bd7a25bb8890bcd93d6e7ac22964408105f9ecc9747e6d085d6d54f"} Oct 30 21:49:28.452005 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:28.451968 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" event={"ID":"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8","Type":"ContainerStarted","Data":"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3"} Oct 30 21:49:28.452462 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:28.452049 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:28.628631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:28.619945 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" podStartSLOduration=2.7728044819999997 podStartE2EDuration="3.619923056s" totalImagesPullingTime="847.118574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:28.592433115 +0000 UTC m=+1593.266479650" watchObservedRunningTime="2026-10-30 21:49:28.619923056 +0000 UTC m=+1593.293969592" Oct 30 21:49:34.464174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:34.464143 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:49:35.397040 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:49:35.396994 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:49:46.583632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.582764 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:46.583632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.583139 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:46.651631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.650906 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:46.761635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.745704 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:46.885768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.885712 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:46.944635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:46.930731 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 21:49:47.041636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:47.029549 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:47.067634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:47.056702 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:47.657649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:47.655517 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:49:47.657649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:47.656308 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:49:48.017686 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.017027 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 21:49:48.052821 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.035157 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:49:48.084467 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.063038 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 21:49:48.084467 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.063415 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:49:48.179644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.175966 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 21:49:48.199360 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.199325 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 21:49:48.214561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:48.204710 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 21:49:54.908553 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.908512 4525 generic.go:356] "Generic (PLEG): container finished" podID="1b965a1d-b306-4d53-8c21-1912a802742c" containerID="938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0" exitCode=0 Oct 30 21:49:54.909052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.908582 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerDied","Data":"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0"} Oct 30 21:49:54.909104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.909089 4525 scope.go:122] "RemoveContainer" containerID="938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0" Oct 30 21:49:54.910824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.910797 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:54.910893 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.910869 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:54.910928 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:54.910901 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:49:55.918565 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:55.918528 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerStarted","Data":"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93"} Oct 30 21:49:56.165668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:56.165601 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:49:56.166886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:56.166862 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:49:56.166949 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:56.166912 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:49:57.166042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:57.166009 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:49:57.166461 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:57.166062 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:49:57.520170 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:57.520087 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:49:58.166846 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:58.166817 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:49:58.167238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:58.166865 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:49:59.166272 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:59.166236 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:49:59.166489 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:49:59.166296 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:50:00.166177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:00.166141 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:50:00.166586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:00.166208 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:50:01.166198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:01.166166 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.166585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:01.166224 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:50:02.170414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:02.170382 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:50:02.171581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:02.171563 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:50:04.983620 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:04.983572 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 21:50:04.987245 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:04.987226 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.057632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.056980 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71a3d788-openshift-default-dockercfg-6mkwc\"" Oct 30 21:50:05.136313 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.136267 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.136657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.136631 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.136819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.136804 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.136972 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.137088 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.137176 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137309 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.137299 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.137408 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndt7\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.137505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.137494 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.210718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.210683 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 21:50:05.272423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.272211 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.272314 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.273641 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndt7\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.273690 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.273997 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.274071 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.274149 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.274308 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.274341 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.275393 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.275699 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.276343 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.277937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.277438 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.279384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.279359 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.308314 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.337753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.337718 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.373685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.373407 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndt7\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.374024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.373999 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-wfkvs\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:05.407694 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:50:05.407654 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:50:05.600659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:05.600549 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:06.046313 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:06.046278 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f06d1d_ae5a_4f0d_9f27_45436aeb37ce.slice/crio-ac53302410d63225daa7f94371a672eba24adfa82885381fb99d27ded958a6fb WatchSource:0}: Error finding container ac53302410d63225daa7f94371a672eba24adfa82885381fb99d27ded958a6fb: Status 404 returned error can't find the container with id ac53302410d63225daa7f94371a672eba24adfa82885381fb99d27ded958a6fb Oct 30 21:50:06.046729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:06.046468 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 21:50:06.048979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:06.048948 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:06.049041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:06.049033 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:06.049076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:06.049064 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:07.023301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:07.023263 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerStarted","Data":"8b8abf001bcacf4c0a475268b3abd4cca1e04714fd599b9bf077065c60d6b573"} Oct 30 21:50:07.023476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:07.023331 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerStarted","Data":"ac53302410d63225daa7f94371a672eba24adfa82885381fb99d27ded958a6fb"} Oct 30 21:50:07.201303 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:07.201249 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podStartSLOduration=3.20123436 podStartE2EDuration="3.20123436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:07.196943476 +0000 UTC m=+1631.870990013" watchObservedRunningTime="2026-10-30 21:50:07.20123436 +0000 UTC m=+1631.875280896" Oct 30 21:50:07.600703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:07.600671 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:07.606905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:07.606880 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:08.030746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:08.030719 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:08.031739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:08.031721 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:50:17.049792 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:17.049761 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv"] Oct 30 21:50:17.099558 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:17.099527 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752f06e-rzgsv"] Oct 30 21:50:17.477592 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:17.477559 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b47cca-109f-4673-8544-c5ef782641f4" path="/var/lib/kubelet/pods/73b47cca-109f-4673-8544-c5ef782641f4/volumes" Oct 30 21:50:23.040631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:23.031223 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:50:23.040631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:23.031558 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" containerID="cri-o://a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa" gracePeriod=30 Oct 30 21:50:28.329744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.329716 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:50:28.333753 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.333729 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.333883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.333765 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.333883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.333827 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.333883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.333853 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.333993 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334017 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334023 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:28.334184 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334086 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334184 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334157 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334268 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:28.334346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334307 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:28.334407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334340 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\") pod \"95ba8857-1093-48ab-b352-ce3bcfcb129e\" (UID: \"95ba8857-1093-48ab-b352-ce3bcfcb129e\") " Oct 30 21:50:28.334483 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334462 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:28.334547 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.334504 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:28.335250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.335226 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ba8857-1093-48ab-b352-ce3bcfcb129e-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.335378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.335252 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.335378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.335270 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.335378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.335284 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.335378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.335298 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.336575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.336547 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:28.336740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.336639 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "kube-api-access-dbpkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:28.336740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.336599 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:28.336740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.336720 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token" pod "95ba8857-1093-48ab-b352-ce3bcfcb129e" (UID: "95ba8857-1093-48ab-b352-ce3bcfcb129e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:28.436814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.436781 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.436814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.436810 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpkk\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-kube-api-access-dbpkk\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.436814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.436821 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.437052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.436831 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ba8857-1093-48ab-b352-ce3bcfcb129e-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:28.460379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.460345 4525 generic.go:356] "Generic (PLEG): container finished" podID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerID="a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa" exitCode=0 Oct 30 21:50:28.460572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.460461 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" event={"ID":"95ba8857-1093-48ab-b352-ce3bcfcb129e","Type":"ContainerDied","Data":"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa"} Oct 30 21:50:28.460572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.460483 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" Oct 30 21:50:28.460572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.460525 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm" event={"ID":"95ba8857-1093-48ab-b352-ce3bcfcb129e","Type":"ContainerDied","Data":"f668694c81c8cf1a408719435b784026c63a86badc1caafc17e755ab63aea0f7"} Oct 30 21:50:28.460572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.460557 4525 scope.go:122] "RemoveContainer" containerID="a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa" Oct 30 21:50:28.472178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.472156 4525 scope.go:122] "RemoveContainer" containerID="a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa" Oct 30 21:50:28.472475 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:50:28.472450 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa\": container with ID starting with a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa not found: ID does not exist" containerID="a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa" Oct 30 21:50:28.472530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.472487 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa"} err="failed to get container status \"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa\": rpc error: code = NotFound desc = could not find container \"a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa\": container with ID starting with a5200c4a0a5a07c2b02004ce4b886a710ed68f9edca7efac279e3227d538f9aa not found: ID does not exist" Oct 30 21:50:28.633895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.633851 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:50:28.637750 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:28.637723 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-l27qm"] Oct 30 21:50:29.475190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:29.475157 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" path="/var/lib/kubelet/pods/95ba8857-1093-48ab-b352-ce3bcfcb129e/volumes" Oct 30 21:50:31.140630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:31.140537 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:50:35.397465 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:50:35.397423 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:50:39.951454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:39.951239 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:50:39.953477 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:39.953448 4525 cpu_manager.go:414] "containerMap: removing container" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" Oct 30 21:50:39.953682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:39.953664 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" Oct 30 21:50:39.954058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:39.954039 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ba8857-1093-48ab-b352-ce3bcfcb129e" containerName="istio-proxy" Oct 30 21:50:39.963054 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:39.963032 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.001393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.001345 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:50:40.060480 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.054714 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b8709fb3-openshift-default-dockercfg-s4zd5\"" Oct 30 21:50:40.111877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.111842 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.111885 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.111983 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112066 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112251 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112422 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112458 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112536 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112492 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.112593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.112576 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4gt\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.220835 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221031 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221317 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221373 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221405 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221465 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4gt\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221550 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221589 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.221669 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.223565 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.223939 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.227677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.224191 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.254631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.232170 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.254631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.232418 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.254631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.236522 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.268632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.264744 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.323634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.314697 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4gt\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.371633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.327220 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\") pod \"e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:40.582744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:40.582651 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:41.059561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.059530 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:50:41.062120 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:41.062088 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7554552_a966_45ce_a189_7abd23f315db.slice/crio-9af8268f2be5e06269d0024ad03604729a18f736e0cbaea26633bbc3649b978d WatchSource:0}: Error finding container 9af8268f2be5e06269d0024ad03604729a18f736e0cbaea26633bbc3649b978d: Status 404 returned error can't find the container with id 9af8268f2be5e06269d0024ad03604729a18f736e0cbaea26633bbc3649b978d Oct 30 21:50:41.065051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.065010 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:41.065155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.065105 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:41.065214 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.065151 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:41.826553 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.826512 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerStarted","Data":"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece"} Oct 30 21:50:41.826774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:41.826573 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerStarted","Data":"9af8268f2be5e06269d0024ad03604729a18f736e0cbaea26633bbc3649b978d"} Oct 30 21:50:42.583560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:42.583529 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:42.589197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:42.589175 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:42.761076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:42.760974 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podStartSLOduration=3.7609598010000003 podStartE2EDuration="3.760959801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:41.950928645 +0000 UTC m=+1666.624975192" watchObservedRunningTime="2026-10-30 21:50:42.760959801 +0000 UTC m=+1667.435006398" Oct 30 21:50:42.840658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:42.840626 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:42.841949 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:42.841922 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:50:47.086206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.084551 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": read tcp 10.133.0.2:60200->10.133.0.93:15021: read: connection reset by peer" start-of-body= Oct 30 21:50:47.086206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.084732 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": read tcp 10.133.0.2:60200->10.133.0.93:15021: read: connection reset by peer" Oct 30 21:50:47.881351 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.881315 4525 generic.go:356] "Generic (PLEG): container finished" podID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerID="e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa" exitCode=0 Oct 30 21:50:47.881527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.881393 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerDied","Data":"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa"} Oct 30 21:50:47.881961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.881929 4525 scope.go:122] "RemoveContainer" containerID="e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa" Oct 30 21:50:47.883814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.883786 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:47.883885 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.883858 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:47.883921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:47.883889 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:49.171634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.170697 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerStarted","Data":"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6"} Oct 30 21:50:49.188743 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.188708 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:50:49.275631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.257954 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.327643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.324199 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8709fb3-dockercfg-b426r\"" Oct 30 21:50:49.387769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.383879 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8709fb3\"" Oct 30 21:50:49.404635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.401680 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:50:49.413640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.408207 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tjv\" (UniqueName: \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.413640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.408638 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.541561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.541437 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:50:49.548169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.548133 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.654679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.630287 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tjv\" (UniqueName: \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.655229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.576496 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.685630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.555817 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.762114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.762082 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tjv\" (UniqueName: \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\") pod \"payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.765153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.765002 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.826943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.826729 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrlf\" (UniqueName: \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.852852 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.785529 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:50:49.914030 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.913984 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:49.930619 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.930570 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.961585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.930848 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrlf\" (UniqueName: \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.961585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.931965 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:49.961585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.949879 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:50:49.961585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.949953 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:50:49.961585 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.961292 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:50:49.992518 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:49.992469 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrlf\" (UniqueName: \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\") pod \"payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:50.171363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:50.171324 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:50.328013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:50.327980 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:50:50.328782 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:50.328751 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc7c0f7_b233_422b_ad24_2fd18c28000c.slice/crio-69775aa47aadf09bb1cf4504513322c444034a027ec30d3c7df356cc59ebe07b WatchSource:0}: Error finding container 69775aa47aadf09bb1cf4504513322c444034a027ec30d3c7df356cc59ebe07b: Status 404 returned error can't find the container with id 69775aa47aadf09bb1cf4504513322c444034a027ec30d3c7df356cc59ebe07b Oct 30 21:50:50.582742 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:50.582708 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:50:50.582896 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:50.582808 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f97e24_1cbd_40e7_876d_b4e6d77e5606.slice/crio-8fde64a8b86195250c73db4d40a01dcae8c489caf40df4700d754bee6fb0eacb WatchSource:0}: Error finding container 8fde64a8b86195250c73db4d40a01dcae8c489caf40df4700d754bee6fb0eacb: Status 404 returned error can't find the container with id 8fde64a8b86195250c73db4d40a01dcae8c489caf40df4700d754bee6fb0eacb Oct 30 21:50:50.952056 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:50.952015 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:50:50.952346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:50.952315 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:50:51.205180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.205087 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" event={"ID":"9bc7c0f7-b233-422b-ad24-2fd18c28000c","Type":"ContainerStarted","Data":"5fe2df643dc4489ea1b1b4c5539e08c4af35a8770b5c03d42818861a7c1c7cc5"} Oct 30 21:50:51.205180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.205140 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" event={"ID":"9bc7c0f7-b233-422b-ad24-2fd18c28000c","Type":"ContainerStarted","Data":"69775aa47aadf09bb1cf4504513322c444034a027ec30d3c7df356cc59ebe07b"} Oct 30 21:50:51.205180 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.205163 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:50:51.207119 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.207094 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" event={"ID":"d6f97e24-1cbd-40e7-876d-b4e6d77e5606","Type":"ContainerStarted","Data":"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013"} Oct 30 21:50:51.207292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.207142 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" event={"ID":"d6f97e24-1cbd-40e7-876d-b4e6d77e5606","Type":"ContainerStarted","Data":"8fde64a8b86195250c73db4d40a01dcae8c489caf40df4700d754bee6fb0eacb"} Oct 30 21:50:51.207292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.207202 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:50:51.477627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.477518 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" podStartSLOduration=2.477503854 podStartE2EDuration="2.477503854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:51.475293247 +0000 UTC m=+1676.149339784" watchObservedRunningTime="2026-10-30 21:50:51.477503854 +0000 UTC m=+1676.151550391" Oct 30 21:50:51.478936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.478895 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" podStartSLOduration=2.478878842 podStartE2EDuration="2.478878842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:51.351924885 +0000 UTC m=+1676.025971422" watchObservedRunningTime="2026-10-30 21:50:51.478878842 +0000 UTC m=+1676.152925380" Oct 30 21:50:51.956945 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.956907 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:50:51.957156 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:51.956970 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:50:52.398504 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:52.398470 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:50:52.496765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:52.496634 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:50:53.027631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:52.959264 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:50:53.129639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:53.094713 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:50:53.954690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:53.950388 4525 patch_prober.go:29] interesting pod/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:50:53.954690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:53.950447 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 21:50:55.252273 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:55.252246 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:50:55.252739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:55.252703 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:50:55.253707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:55.253692 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:50:57.020874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.019524 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:50:57.051323 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.051290 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.242202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.242170 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-259e81-dockercfg-2dc6q\"" Oct 30 21:50:57.268525 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.266992 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-259e81\"" Oct 30 21:50:57.272798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.272721 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pc4\" (UniqueName: \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.273465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.273438 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.384828 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.384777 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.434106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.434069 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:50:57.434385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.434368 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:50:57.440785 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.440746 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pc4\" (UniqueName: \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.443230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.443193 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.443546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.440759 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:50:57.469824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.448552 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.478632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.470766 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" podUID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" containerName="payload-processing" containerID="cri-o://151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a" gracePeriod=25 Oct 30 21:50:57.522981 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.522890 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pc4\" (UniqueName: \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\") pod \"payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.573134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.571424 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" podUID="2012a45a-5810-4c8f-926f-a65be0e22ffc" containerName="payload-pre-processing" containerID="cri-o://3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34" gracePeriod=25 Oct 30 21:50:57.594816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.588915 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:57.660891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.659723 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdcp\" (UniqueName: \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.660891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.659950 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.761331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.761297 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.761593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.761563 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdcp\" (UniqueName: \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.762206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.762181 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.825217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.825148 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdcp\" (UniqueName: \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\") pod \"payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.890053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.890025 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:50:57.907314 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.907277 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:50:57.944728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.944697 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:50:57.964696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.964641 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\") pod \"2012a45a-5810-4c8f-926f-a65be0e22ffc\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " Oct 30 21:50:57.964831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.964803 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\" (UniqueName: \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\") pod \"2012a45a-5810-4c8f-926f-a65be0e22ffc\" (UID: \"2012a45a-5810-4c8f-926f-a65be0e22ffc\") " Oct 30 21:50:57.965330 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.965013 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume" pod "2012a45a-5810-4c8f-926f-a65be0e22ffc" (UID: "2012a45a-5810-4c8f-926f-a65be0e22ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:57.965953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.965928 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2012a45a-5810-4c8f-926f-a65be0e22ffc-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:57.967253 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.967231 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j" pod "2012a45a-5810-4c8f-926f-a65be0e22ffc" (UID: "2012a45a-5810-4c8f-926f-a65be0e22ffc"). InnerVolumeSpecName "kube-api-access-4sd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:57.995697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:57.994949 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:50:58.032723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.032551 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:50:58.056139 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:58.056111 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844df8d5_fb08_4b72_b2b5_a087c99a4f1d.slice/crio-b1d3ccffe3cb484009cd8be6081d24da06781b6da0573e5ad1e77bcd0e1d596c WatchSource:0}: Error finding container b1d3ccffe3cb484009cd8be6081d24da06781b6da0573e5ad1e77bcd0e1d596c: Status 404 returned error can't find the container with id b1d3ccffe3cb484009cd8be6081d24da06781b6da0573e5ad1e77bcd0e1d596c Oct 30 21:50:58.056400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.056380 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:50:58.066758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.066734 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\") pod \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " Oct 30 21:50:58.067058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.067039 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\" (UniqueName: \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\") pod \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\" (UID: \"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043\") " Oct 30 21:50:58.067095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.067059 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume" pod "4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" (UID: "4ab8424a-6792-4bd5-a4c5-79fd8f8e8043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:58.067780 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.067758 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sd8j\" (UniqueName: \"kubernetes.io/projected/2012a45a-5810-4c8f-926f-a65be0e22ffc-kube-api-access-4sd8j\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.067878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.067785 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.069310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.069289 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk" pod "4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" (UID: "4ab8424a-6792-4bd5-a4c5-79fd8f8e8043"). InnerVolumeSpecName "kube-api-access-86stk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:58.170238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.170207 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86stk\" (UniqueName: \"kubernetes.io/projected/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043-kube-api-access-86stk\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:58.281956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.281918 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:50:58.282435 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:50:58.282395 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f002bb9_dd36_4bdb_8284_78cfd2618a71.slice/crio-cadea8b2ff3a050b3cb983e1bf34d62aef6df2e62dd1eb93966fd9bbca7bee10 WatchSource:0}: Error finding container cadea8b2ff3a050b3cb983e1bf34d62aef6df2e62dd1eb93966fd9bbca7bee10: Status 404 returned error can't find the container with id cadea8b2ff3a050b3cb983e1bf34d62aef6df2e62dd1eb93966fd9bbca7bee10 Oct 30 21:50:58.636962 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.636922 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" event={"ID":"844df8d5-fb08-4b72-b2b5-a087c99a4f1d","Type":"ContainerStarted","Data":"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6"} Oct 30 21:50:58.637229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.637207 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" event={"ID":"844df8d5-fb08-4b72-b2b5-a087c99a4f1d","Type":"ContainerStarted","Data":"b1d3ccffe3cb484009cd8be6081d24da06781b6da0573e5ad1e77bcd0e1d596c"} Oct 30 21:50:58.638634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.638592 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:50:58.644888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.642942 4525 generic.go:356] "Generic (PLEG): container finished" podID="2012a45a-5810-4c8f-926f-a65be0e22ffc" containerID="3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34" exitCode=0 Oct 30 21:50:58.644888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.643121 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" event={"ID":"2012a45a-5810-4c8f-926f-a65be0e22ffc","Type":"ContainerDied","Data":"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34"} Oct 30 21:50:58.644888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.643165 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" event={"ID":"2012a45a-5810-4c8f-926f-a65be0e22ffc","Type":"ContainerDied","Data":"9196630f55d5008aff5de48c38f3381859644d43a589fadfd6808228769a7c94"} Oct 30 21:50:58.644888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.643193 4525 scope.go:122] "RemoveContainer" containerID="3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34" Oct 30 21:50:58.644888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.643366 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt" Oct 30 21:50:58.678632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.674710 4525 generic.go:356] "Generic (PLEG): container finished" podID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" containerID="151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a" exitCode=0 Oct 30 21:50:58.678632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.674871 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" event={"ID":"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043","Type":"ContainerDied","Data":"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a"} Oct 30 21:50:58.678632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.674932 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" event={"ID":"4ab8424a-6792-4bd5-a4c5-79fd8f8e8043","Type":"ContainerDied","Data":"19aa87a9b8c744992b4c8acfa5912bb4ff63820413ed65ed0013250b84629628"} Oct 30 21:50:58.678632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.675030 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k" Oct 30 21:50:58.685746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.685701 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" event={"ID":"0f002bb9-dd36-4bdb-8284-78cfd2618a71","Type":"ContainerStarted","Data":"cadea8b2ff3a050b3cb983e1bf34d62aef6df2e62dd1eb93966fd9bbca7bee10"} Oct 30 21:50:58.698216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.698192 4525 scope.go:122] "RemoveContainer" containerID="3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34" Oct 30 21:50:58.699451 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:50:58.699424 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34\": container with ID starting with 3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34 not found: ID does not exist" containerID="3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34" Oct 30 21:50:58.699707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.699677 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34"} err="failed to get container status \"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34\": rpc error: code = NotFound desc = could not find container \"3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34\": container with ID starting with 3fd1111ee2d580ed41f567142f131052ac91d9babe4a58f41e56a31ed6518a34 not found: ID does not exist" Oct 30 21:50:58.700644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.700626 4525 scope.go:122] "RemoveContainer" containerID="151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a" Oct 30 21:50:58.725340 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.725318 4525 scope.go:122] "RemoveContainer" containerID="151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a" Oct 30 21:50:58.725759 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:50:58.725739 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a\": container with ID starting with 151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a not found: ID does not exist" containerID="151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a" Oct 30 21:50:58.725848 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.725771 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a"} err="failed to get container status \"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a\": rpc error: code = NotFound desc = could not find container \"151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a\": container with ID starting with 151ae6dd91bc276fe2954148343a180bd8675ec1c22c2322198d9b3d966c4f6a not found: ID does not exist" Oct 30 21:50:58.764590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.764540 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" podStartSLOduration=2.764523899 podStartE2EDuration="2.764523899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:58.763890559 +0000 UTC m=+1683.437937098" watchObservedRunningTime="2026-10-30 21:50:58.764523899 +0000 UTC m=+1683.438570436" Oct 30 21:50:58.878974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.878940 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:50:58.882955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:58.882928 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fd496e14-6694796c5-5znvt"] Oct 30 21:50:59.048365 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.037410 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:50:59.048365 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.044467 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fd496e14-5b7c7857f8-csb7k"] Oct 30 21:50:59.668030 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.667990 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2012a45a-5810-4c8f-926f-a65be0e22ffc" path="/var/lib/kubelet/pods/2012a45a-5810-4c8f-926f-a65be0e22ffc/volumes" Oct 30 21:50:59.803423 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.803384 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" path="/var/lib/kubelet/pods/4ab8424a-6792-4bd5-a4c5-79fd8f8e8043/volumes" Oct 30 21:50:59.990694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.989871 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" event={"ID":"0f002bb9-dd36-4bdb-8284-78cfd2618a71","Type":"ContainerStarted","Data":"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106"} Oct 30 21:50:59.991012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:50:59.990944 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:51:00.117138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:00.117083 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" podStartSLOduration=3.117066083 podStartE2EDuration="3.117066083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:00.110384044 +0000 UTC m=+1684.784430580" watchObservedRunningTime="2026-10-30 21:51:00.117066083 +0000 UTC m=+1684.791112622" Oct 30 21:51:02.218976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:02.218942 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:51:02.219456 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:02.219014 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:51:02.907650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:02.907557 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" podUID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" containerName="payload-processing" containerID="cri-o://1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228" gracePeriod=25 Oct 30 21:51:03.032961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.032920 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" podUID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" containerName="payload-pre-processing" containerID="cri-o://bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80" gracePeriod=25 Oct 30 21:51:03.201546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.201215 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:51:03.297062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.297030 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\") pod \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " Oct 30 21:51:03.297539 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.297353 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\" (UniqueName: \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\") pod \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\" (UID: \"ea29bba8-9004-4a64-8502-5cce1cbe9eac\") " Oct 30 21:51:03.297539 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.297487 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume" pod "ea29bba8-9004-4a64-8502-5cce1cbe9eac" (UID: "ea29bba8-9004-4a64-8502-5cce1cbe9eac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:03.298131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.298098 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea29bba8-9004-4a64-8502-5cce1cbe9eac-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:03.300020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.299988 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj" pod "ea29bba8-9004-4a64-8502-5cce1cbe9eac" (UID: "ea29bba8-9004-4a64-8502-5cce1cbe9eac"). InnerVolumeSpecName "kube-api-access-l6dnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:03.324397 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.324081 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:51:03.398794 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.398766 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\" (UniqueName: \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\") pod \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " Oct 30 21:51:03.398961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.398870 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\") pod \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\" (UID: \"a23bc062-9f29-4bc4-9dd9-69a889c372e4\") " Oct 30 21:51:03.399270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.399238 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume" pod "a23bc062-9f29-4bc4-9dd9-69a889c372e4" (UID: "a23bc062-9f29-4bc4-9dd9-69a889c372e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:03.399544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.399527 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a23bc062-9f29-4bc4-9dd9-69a889c372e4-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:03.399590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.399550 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dnj\" (UniqueName: \"kubernetes.io/projected/ea29bba8-9004-4a64-8502-5cce1cbe9eac-kube-api-access-l6dnj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:03.401045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.401023 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p" pod "a23bc062-9f29-4bc4-9dd9-69a889c372e4" (UID: "a23bc062-9f29-4bc4-9dd9-69a889c372e4"). InnerVolumeSpecName "kube-api-access-9kg7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:03.501971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:03.501939 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kg7p\" (UniqueName: \"kubernetes.io/projected/a23bc062-9f29-4bc4-9dd9-69a889c372e4-kube-api-access-9kg7p\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:04.020556 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.020523 4525 generic.go:356] "Generic (PLEG): container finished" podID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" containerID="1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228" exitCode=0 Oct 30 21:51:04.020761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.020576 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" Oct 30 21:51:04.020761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.020573 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" event={"ID":"ea29bba8-9004-4a64-8502-5cce1cbe9eac","Type":"ContainerDied","Data":"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228"} Oct 30 21:51:04.020761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.020673 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x" event={"ID":"ea29bba8-9004-4a64-8502-5cce1cbe9eac","Type":"ContainerDied","Data":"81b72613ee3093fafb2a8d38a0afdeafbd7d790c97a4f8988b3b65ce2b12f391"} Oct 30 21:51:04.020761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.020704 4525 scope.go:122] "RemoveContainer" containerID="1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228" Oct 30 21:51:04.022207 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.022182 4525 generic.go:356] "Generic (PLEG): container finished" podID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" containerID="bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80" exitCode=0 Oct 30 21:51:04.022321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.022211 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" event={"ID":"a23bc062-9f29-4bc4-9dd9-69a889c372e4","Type":"ContainerDied","Data":"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80"} Oct 30 21:51:04.022321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.022248 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" Oct 30 21:51:04.022321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.022259 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg" event={"ID":"a23bc062-9f29-4bc4-9dd9-69a889c372e4","Type":"ContainerDied","Data":"fa359938f011528f3a7f5e802717cb04c821e903d3ecca807ff88061a3acdb8b"} Oct 30 21:51:04.031301 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.031284 4525 scope.go:122] "RemoveContainer" containerID="1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228" Oct 30 21:51:04.031728 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:04.031584 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228\": container with ID starting with 1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228 not found: ID does not exist" containerID="1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228" Oct 30 21:51:04.031728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.031634 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228"} err="failed to get container status \"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228\": rpc error: code = NotFound desc = could not find container \"1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228\": container with ID starting with 1fdda4fabac60b2a93e7808fb72ee5d01ae356dfb625ccb944854187bc4dc228 not found: ID does not exist" Oct 30 21:51:04.031728 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.031653 4525 scope.go:122] "RemoveContainer" containerID="bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80" Oct 30 21:51:04.043184 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.043160 4525 scope.go:122] "RemoveContainer" containerID="bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80" Oct 30 21:51:04.043531 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:04.043510 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80\": container with ID starting with bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80 not found: ID does not exist" containerID="bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80" Oct 30 21:51:04.043643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.043540 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80"} err="failed to get container status \"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80\": rpc error: code = NotFound desc = could not find container \"bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80\": container with ID starting with bdf37aeefb031c195e279acc53ff1db720cef706f905441e6f46c52dde0a7f80 not found: ID does not exist" Oct 30 21:51:04.196892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.196789 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:51:04.237877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.237840 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-21084e54-5785648f7d-mrq9x"] Oct 30 21:51:04.306362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.306320 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:51:04.313422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:04.313395 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-21084e54-5897b47d55-xjrxg"] Oct 30 21:51:05.398844 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:05.398805 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:51:05.477970 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:05.477934 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" path="/var/lib/kubelet/pods/a23bc062-9f29-4bc4-9dd9-69a889c372e4/volumes" Oct 30 21:51:05.478494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:05.478477 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" path="/var/lib/kubelet/pods/ea29bba8-9004-4a64-8502-5cce1cbe9eac/volumes" Oct 30 21:51:08.044639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.044589 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:51:08.045041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.044821 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" containerID="cri-o://617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886" gracePeriod=30 Oct 30 21:51:08.094062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.094029 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:51:08.094281 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.094240 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" containerID="cri-o://845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7" gracePeriod=30 Oct 30 21:51:08.097848 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.097822 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:51:08.098029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.098009 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" containerID="cri-o://866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335" gracePeriod=30 Oct 30 21:51:08.143662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.143626 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:51:08.143881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.143836 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" containerID="cri-o://dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02" gracePeriod=30 Oct 30 21:51:08.512070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.512039 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:51:08.512296 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:08.512272 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" containerID="cri-o://044c39e5a7a957fba0cd1437be163c4fb294506b12fffa9e1eb89c1116dbfdc6" gracePeriod=30 Oct 30 21:51:10.817935 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:10.817898 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:10.818338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:10.817954 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:11.001748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:11.001721 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:51:12.009159 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:12.009128 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:51:12.410697 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:12.410657 4525 patch_prober.go:29] interesting pod/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:12.410869 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:12.410725 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:13.408625 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.408560 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494079 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494244 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494393 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494436 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494487 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494517 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494568 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494595 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.494649 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\") pod \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\" (UID: \"5d25c6f8-5a41-4a27-bee2-4181836d09ad\") " Oct 30 21:51:13.498711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.496312 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.498711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.496512 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.498711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.496720 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:13.498711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.498281 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.499370 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.499334 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.502843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.501863 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "kube-api-access-mbwdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.502843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.502548 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.502843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.502740 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:13.509813 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.509767 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy" pod "5d25c6f8-5a41-4a27-bee2-4181836d09ad" (UID: "5d25c6f8-5a41-4a27-bee2-4181836d09ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.520802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.520730 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:51:13.530177 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.530151 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:51:13.585218 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.585189 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.599813 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.599892 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.599951 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600027 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600091 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600123 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600195 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600227 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600262 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600314 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600392 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600522 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600580 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600647 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600681 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600762 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\") pod \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\" (UID: \"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600862 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.601638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.600920 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\") pod \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\" (UID: \"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab\") " Oct 30 21:51:13.603190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603085 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwdx\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-kube-api-access-mbwdx\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.603190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603115 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.603190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603129 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.603190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603141 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.604174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603868 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.604174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603898 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.604174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603916 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.604174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603929 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d25c6f8-5a41-4a27-bee2-4181836d09ad-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.604174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.603942 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d25c6f8-5a41-4a27-bee2-4181836d09ad-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.604630 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.605852 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606135 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606170 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606299 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606370 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606393 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606793 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606737 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.606994 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.607013 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.609353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.609141 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.610014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.609717 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:13.610014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.609852 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.610014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.609875 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "kube-api-access-v6bhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.610712 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.610672 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.611761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.611732 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6" pod "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" (UID: "bb5fd30d-e8e0-42d4-950c-9ba9fba264ab"). InnerVolumeSpecName "kube-api-access-d9mw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.612021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.611999 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy" pod "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" (UID: "a83e39ca-ed22-41a6-a062-b58c4b1a7fa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.705046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.704962 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705056 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705282 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705219 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705281 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705375 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705324 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705375 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705354 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705417 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.705488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705456 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.705680 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\") pod \"fb555670-db7e-432f-b0b3-5dd8150a09f8\" (UID: \"fb555670-db7e-432f-b0b3-5dd8150a09f8\") " Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706556 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706589 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706621 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706635 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706656 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706672 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706685 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706698 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706712 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706725 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706740 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mw6\" (UniqueName: \"kubernetes.io/projected/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-kube-api-access-d9mw6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706754 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706769 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6bhb\" (UniqueName: \"kubernetes.io/projected/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-kube-api-access-v6bhb\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706783 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706797 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706810 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706836 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.707269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.706850 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.715577 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.715837 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.715886 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.715921 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.716251 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.717587 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.721000 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "kube-api-access-z7f7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.721791 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:13.724527 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.722877 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy" pod "fb555670-db7e-432f-b0b3-5dd8150a09f8" (UID: "fb555670-db7e-432f-b0b3-5dd8150a09f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:13.808247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808205 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808238 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb555670-db7e-432f-b0b3-5dd8150a09f8-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808247 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808258 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808266 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808276 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808284 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7f7t\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-kube-api-access-z7f7t\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808292 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:13.808542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:13.808300 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb555670-db7e-432f-b0b3-5dd8150a09f8-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.105125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.105084 4525 generic.go:356] "Generic (PLEG): container finished" podID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerID="845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7" exitCode=0 Oct 30 21:51:14.105295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.105179 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" event={"ID":"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7","Type":"ContainerDied","Data":"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7"} Oct 30 21:51:14.105295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.105211 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" Oct 30 21:51:14.105295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.105263 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl" event={"ID":"a83e39ca-ed22-41a6-a062-b58c4b1a7fa7","Type":"ContainerDied","Data":"de1cac3f3b6199dde79a6ccd3b464d3a4838800b0d6b4b79d4ed5582ebbeb34d"} Oct 30 21:51:14.105460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.105297 4525 scope.go:122] "RemoveContainer" containerID="845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7" Oct 30 21:51:14.108212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.107270 4525 generic.go:356] "Generic (PLEG): container finished" podID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerID="044c39e5a7a957fba0cd1437be163c4fb294506b12fffa9e1eb89c1116dbfdc6" exitCode=0 Oct 30 21:51:14.108708 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.108679 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" event={"ID":"536f6deb-1e4b-4616-af3a-097a6a5acfcc","Type":"ContainerDied","Data":"044c39e5a7a957fba0cd1437be163c4fb294506b12fffa9e1eb89c1116dbfdc6"} Oct 30 21:51:14.113019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.112990 4525 generic.go:356] "Generic (PLEG): container finished" podID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerID="dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02" exitCode=0 Oct 30 21:51:14.113138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.113070 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" Oct 30 21:51:14.113138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.113112 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" event={"ID":"fb555670-db7e-432f-b0b3-5dd8150a09f8","Type":"ContainerDied","Data":"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02"} Oct 30 21:51:14.113274 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.113162 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b" event={"ID":"fb555670-db7e-432f-b0b3-5dd8150a09f8","Type":"ContainerDied","Data":"590be0f206d50a503a3815cb96cb0cf67268785a7588c48c8d8e4dddb123ccc7"} Oct 30 21:51:14.115042 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.114964 4525 generic.go:356] "Generic (PLEG): container finished" podID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerID="866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335" exitCode=0 Oct 30 21:51:14.115122 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.115081 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" event={"ID":"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab","Type":"ContainerDied","Data":"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335"} Oct 30 21:51:14.115175 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.115121 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" event={"ID":"bb5fd30d-e8e0-42d4-950c-9ba9fba264ab","Type":"ContainerDied","Data":"7a199a872fd88d82819ceb37f83d45c790ff194f5282a1ab40f26c964ebfc550"} Oct 30 21:51:14.115175 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.115127 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx" Oct 30 21:51:14.120847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.117902 4525 generic.go:356] "Generic (PLEG): container finished" podID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerID="617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886" exitCode=0 Oct 30 21:51:14.120847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.118018 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" event={"ID":"5d25c6f8-5a41-4a27-bee2-4181836d09ad","Type":"ContainerDied","Data":"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886"} Oct 30 21:51:14.120847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.118055 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" event={"ID":"5d25c6f8-5a41-4a27-bee2-4181836d09ad","Type":"ContainerDied","Data":"9f27a2f0a8951242187477599afb9a7f6819901f89ce86716168caf6056766d2"} Oct 30 21:51:14.120847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.118138 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr" Oct 30 21:51:14.134882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.134712 4525 scope.go:122] "RemoveContainer" containerID="845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7" Oct 30 21:51:14.136648 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:14.136587 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7\": container with ID starting with 845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7 not found: ID does not exist" containerID="845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7" Oct 30 21:51:14.136756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.136659 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7"} err="failed to get container status \"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7\": rpc error: code = NotFound desc = could not find container \"845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7\": container with ID starting with 845b47bbc90729bc2b0978408109fa47afc2487b98ff533de431d9e5f6c267f7 not found: ID does not exist" Oct 30 21:51:14.136756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.136678 4525 scope.go:122] "RemoveContainer" containerID="dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02" Oct 30 21:51:14.150043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.150023 4525 scope.go:122] "RemoveContainer" containerID="dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02" Oct 30 21:51:14.150522 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:14.150373 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02\": container with ID starting with dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02 not found: ID does not exist" containerID="dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02" Oct 30 21:51:14.150522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.150417 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02"} err="failed to get container status \"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02\": rpc error: code = NotFound desc = could not find container \"dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02\": container with ID starting with dac112c1b01c56905faad2f142a6dc95bdbf076c6acf2f8085f78d4aacec4c02 not found: ID does not exist" Oct 30 21:51:14.150522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.150447 4525 scope.go:122] "RemoveContainer" containerID="866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335" Oct 30 21:51:14.207140 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.206923 4525 scope.go:122] "RemoveContainer" containerID="866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335" Oct 30 21:51:14.207387 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:14.207362 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335\": container with ID starting with 866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335 not found: ID does not exist" containerID="866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335" Oct 30 21:51:14.207442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.207399 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335"} err="failed to get container status \"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335\": rpc error: code = NotFound desc = could not find container \"866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335\": container with ID starting with 866e2e237aa495b41275af85efff7c00ce1164b6c47955d365e8e12fab6cf335 not found: ID does not exist" Oct 30 21:51:14.207442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.207424 4525 scope.go:122] "RemoveContainer" containerID="617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886" Oct 30 21:51:14.220692 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.220674 4525 scope.go:122] "RemoveContainer" containerID="617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886" Oct 30 21:51:14.221095 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:14.221060 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886\": container with ID starting with 617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886 not found: ID does not exist" containerID="617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886" Oct 30 21:51:14.221194 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.221100 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886"} err="failed to get container status \"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886\": rpc error: code = NotFound desc = could not find container \"617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886\": container with ID starting with 617667e5013f6be0b7710b3e267cb33fb60b67a497898c346d776d56a3348886 not found: ID does not exist" Oct 30 21:51:14.255450 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.255423 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:51:14.284978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.284942 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:51:14.295700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.290881 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-z8wbr"] Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434632 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434742 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434787 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434832 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434870 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.434949 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.435061 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.435283 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.435630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.435343 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\") pod \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\" (UID: \"536f6deb-1e4b-4616-af3a-097a6a5acfcc\") " Oct 30 21:51:14.438464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.437827 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:14.438464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.438140 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:14.439070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.439043 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:14.439710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.439688 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:14.450656 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.450585 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:14.466993 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.466449 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:51:14.475669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.475639 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "kube-api-access-htg4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:14.475833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.475789 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:14.476095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.476071 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-bcrbl"] Oct 30 21:51:14.476263 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.476240 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:14.476796 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.476761 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo" pod "536f6deb-1e4b-4616-af3a-097a6a5acfcc" (UID: "536f6deb-1e4b-4616-af3a-097a6a5acfcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548356 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548393 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548409 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548421 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548434 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548447 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548460 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htg4d\" (UniqueName: \"kubernetes.io/projected/536f6deb-1e4b-4616-af3a-097a6a5acfcc-kube-api-access-htg4d\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548474 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/536f6deb-1e4b-4616-af3a-097a6a5acfcc-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.551687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.548497 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/536f6deb-1e4b-4616-af3a-097a6a5acfcc-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:14.568966 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.568823 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:51:14.576973 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.572496 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-xkkvx"] Oct 30 21:51:14.661325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.661284 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:51:14.664910 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.664884 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1de75a-openshift-default-5dfc4949f5-b8h2b"] Oct 30 21:51:14.878121 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878090 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:51:14.878884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878866 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" Oct 30 21:51:14.878884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878885 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878906 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" containerName="payload-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878912 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" containerName="payload-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878929 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" containerName="payload-pre-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878937 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" containerName="payload-pre-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878952 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878957 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878963 4525 cpu_manager.go:414] "containerMap: removing container" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878967 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878982 4525 cpu_manager.go:414] "containerMap: removing container" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878986 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878993 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" containerName="payload-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.878997 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" containerName="payload-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879008 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2012a45a-5810-4c8f-926f-a65be0e22ffc" containerName="payload-pre-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879012 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2012a45a-5810-4c8f-926f-a65be0e22ffc" containerName="payload-pre-processing" Oct 30 21:51:14.879015 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879021 4525 cpu_manager.go:414] "containerMap: removing container" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879025 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879117 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23bc062-9f29-4bc4-9dd9-69a889c372e4" containerName="payload-pre-processing" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879125 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879131 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879138 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879142 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea29bba8-9004-4a64-8502-5cce1cbe9eac" containerName="payload-processing" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879149 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2012a45a-5810-4c8f-926f-a65be0e22ffc" containerName="payload-pre-processing" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879154 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879163 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" containerName="istio-proxy" Oct 30 21:51:14.879425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.879168 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab8424a-6792-4bd5-a4c5-79fd8f8e8043" containerName="payload-processing" Oct 30 21:51:14.883032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.883016 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:14.936094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.936056 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71a3d788-serving-cert\"" Oct 30 21:51:14.941328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:14.941307 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:51:15.056590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.056553 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.056758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.056635 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.056845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.056822 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6h4t\" (UniqueName: \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.057016 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.056999 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.171676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.159825 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6h4t\" (UniqueName: \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.171676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.159994 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.171676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.160223 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.171676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.160271 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.171676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.171244 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.179724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.175381 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.179724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.177905 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.181906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.181003 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" event={"ID":"536f6deb-1e4b-4616-af3a-097a6a5acfcc","Type":"ContainerDied","Data":"20e83b05e2a3821be99b98a027c8edb3874435f59cd5cc0d7bf85f021e778fb7"} Oct 30 21:51:15.181906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.181138 4525 scope.go:122] "RemoveContainer" containerID="044c39e5a7a957fba0cd1437be163c4fb294506b12fffa9e1eb89c1116dbfdc6" Oct 30 21:51:15.181906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.181308 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb" Oct 30 21:51:15.263638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.254510 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6h4t\" (UniqueName: \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.398730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.398695 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:51:15.411687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.411651 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-gr2tb"] Oct 30 21:51:15.475693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.475622 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536f6deb-1e4b-4616-af3a-097a6a5acfcc" path="/var/lib/kubelet/pods/536f6deb-1e4b-4616-af3a-097a6a5acfcc/volumes" Oct 30 21:51:15.476125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.476111 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d25c6f8-5a41-4a27-bee2-4181836d09ad" path="/var/lib/kubelet/pods/5d25c6f8-5a41-4a27-bee2-4181836d09ad/volumes" Oct 30 21:51:15.476562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.476548 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83e39ca-ed22-41a6-a062-b58c4b1a7fa7" path="/var/lib/kubelet/pods/a83e39ca-ed22-41a6-a062-b58c4b1a7fa7/volumes" Oct 30 21:51:15.477326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.477311 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5fd30d-e8e0-42d4-950c-9ba9fba264ab" path="/var/lib/kubelet/pods/bb5fd30d-e8e0-42d4-950c-9ba9fba264ab/volumes" Oct 30 21:51:15.477784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.477771 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb555670-db7e-432f-b0b3-5dd8150a09f8" path="/var/lib/kubelet/pods/fb555670-db7e-432f-b0b3-5dd8150a09f8/volumes" Oct 30 21:51:15.509398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.509367 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:15.895913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.895883 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:51:15.896065 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:15.895940 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b931a0_fa52_4f2a_a309_b21c939a70d7.slice/crio-2e1e2708929e4311e1c9eaab802e6039bed5aec75e246a754403abf1375dc96b WatchSource:0}: Error finding container 2e1e2708929e4311e1c9eaab802e6039bed5aec75e246a754403abf1375dc96b: Status 404 returned error can't find the container with id 2e1e2708929e4311e1c9eaab802e6039bed5aec75e246a754403abf1375dc96b Oct 30 21:51:15.897366 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:15.897350 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:51:16.260327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:16.260237 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" event={"ID":"a1b931a0-fa52-4f2a-a309-b21c939a70d7","Type":"ContainerStarted","Data":"2e1e2708929e4311e1c9eaab802e6039bed5aec75e246a754403abf1375dc96b"} Oct 30 21:51:17.288250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:17.284901 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" event={"ID":"a1b931a0-fa52-4f2a-a309-b21c939a70d7","Type":"ContainerStarted","Data":"91196bd860c9a395592eeaffead0447644d67c181994e2c7b94897b7df16b0bf"} Oct 30 21:51:17.291676 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:17.289834 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:17.407881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:17.407829 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" podStartSLOduration=2.573083345 podStartE2EDuration="3.407812994s" totalImagesPullingTime="834.729649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:17.407072698 +0000 UTC m=+1702.081119273" watchObservedRunningTime="2026-10-30 21:51:17.407812994 +0000 UTC m=+1702.081859574" Oct 30 21:51:21.757729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.757679 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 21:51:21.830681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.829339 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932038 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932122 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932209 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932219 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932301 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932475 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932587 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932731 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932789 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.941471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.932827 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5h9\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:21.953632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:21.948096 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-1a415f-openshift-default-dockercfg-rrxhc\"" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106006 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106119 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106163 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106270 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106345 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106394 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106442 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5h9\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.106690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.106558 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.109541 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.110691 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.111410 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.113325 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.116292 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.127239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.118238 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.131669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.129712 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.151844 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.146381 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.185631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.183082 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5h9\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.194754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.186627 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\") pod \"e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.284032 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.268285 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:22.734410 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:22.734380 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf340ba07_fb78_4e39_8be7_c1fd6deac8ee.slice/crio-d33191bb7573c310b0150708cfc74d8fc727f57a9987cabd85838590e362fb6f WatchSource:0}: Error finding container d33191bb7573c310b0150708cfc74d8fc727f57a9987cabd85838590e362fb6f: Status 404 returned error can't find the container with id d33191bb7573c310b0150708cfc74d8fc727f57a9987cabd85838590e362fb6f Oct 30 21:51:22.734529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.734464 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 21:51:22.737192 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.737161 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:22.737239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.737227 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:22.737279 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:22.737255 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:23.332290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:23.332256 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:51:23.379581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:23.379542 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerStarted","Data":"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0"} Oct 30 21:51:23.379760 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:23.379595 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerStarted","Data":"d33191bb7573c310b0150708cfc74d8fc727f57a9987cabd85838590e362fb6f"} Oct 30 21:51:23.586388 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:23.586284 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podStartSLOduration=2.586269095 podStartE2EDuration="2.586269095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.584726875 +0000 UTC m=+1708.258773413" watchObservedRunningTime="2026-10-30 21:51:23.586269095 +0000 UTC m=+1708.260315631" Oct 30 21:51:24.269914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.269882 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:24.276247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.276223 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:24.386204 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.386173 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:24.387444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.387421 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:51:24.800776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.800745 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:51:24.800987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:24.800966 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" podUID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" containerName="maas-api" containerID="cri-o://3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3" gracePeriod=30 Oct 30 21:51:25.199346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.199318 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:51:25.239077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.238102 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") pod \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " Oct 30 21:51:25.239077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.238268 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\" (UniqueName: \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\") pod \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " Oct 30 21:51:25.239077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.238496 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\") pod \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " Oct 30 21:51:25.239077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.238544 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\") pod \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\" (UID: \"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8\") " Oct 30 21:51:25.241270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.240702 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca" pod "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" (UID: "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:25.243227 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.241586 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.266649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.263885 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q" pod "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" (UID: "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8"). InnerVolumeSpecName "kube-api-access-bw26q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:25.284848 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.281030 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls" pod "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" (UID: "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:25.292689 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.290795 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs" pod "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" (UID: "9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:25.360633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.351830 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.360633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.351920 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.360633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.351938 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw26q\" (UniqueName: \"kubernetes.io/projected/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8-kube-api-access-bw26q\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:25.423687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.419387 4525 generic.go:356] "Generic (PLEG): container finished" podID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" containerID="3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3" exitCode=0 Oct 30 21:51:25.423687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.419492 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" Oct 30 21:51:25.457629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.419509 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" event={"ID":"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8","Type":"ContainerDied","Data":"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3"} Oct 30 21:51:25.457629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.457378 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb" event={"ID":"9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8","Type":"ContainerDied","Data":"492484dc3bd7a25bb8890bcd93d6e7ac22964408105f9ecc9747e6d085d6d54f"} Oct 30 21:51:25.457629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.457423 4525 scope.go:122] "RemoveContainer" containerID="3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3" Oct 30 21:51:25.506333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.502101 4525 scope.go:122] "RemoveContainer" containerID="3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3" Oct 30 21:51:25.511079 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:25.510398 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3\": container with ID starting with 3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3 not found: ID does not exist" containerID="3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3" Oct 30 21:51:25.511079 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.510473 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3"} err="failed to get container status \"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3\": rpc error: code = NotFound desc = could not find container \"3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3\": container with ID starting with 3cce8373cda33cd82197a764247577009ee02e1f036ca2c15ba9e21eda6fe4c3 not found: ID does not exist" Oct 30 21:51:25.629465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.629434 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:51:25.634418 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:25.634391 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-21084e54-ff876c4c7-b2jpb"] Oct 30 21:51:27.283793 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:27.283757 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:51:27.284287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:27.284053 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" containerID="cri-o://b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93" gracePeriod=30 Oct 30 21:51:27.617819 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:27.617781 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" path="/var/lib/kubelet/pods/9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8/volumes" Oct 30 21:51:28.599933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.599892 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:51:28.600922 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.600903 4525 cpu_manager.go:414] "containerMap: removing container" podUID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" containerName="maas-api" Oct 30 21:51:28.600922 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.600921 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" containerName="maas-api" Oct 30 21:51:28.601116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.601085 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db1caa2-0e9c-413a-b6e1-1e24c9d4dbf8" containerName="maas-api" Oct 30 21:51:28.621958 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.620362 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:28.693782 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.693747 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:51:28.810660 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:28.804753 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d6e0be-openshift-default-dockercfg-dfgqp\"" Oct 30 21:51:29.039473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039430 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.039473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039480 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039575 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039623 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039654 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5m4z\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039745 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039795 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039869 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.040212 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.039909 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.143916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.143855 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.145201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.145175 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.145652 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.145525 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.145808 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.145790 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146014 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.145996 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146120 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146235 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5m4z\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146414 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146710 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146024 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146249 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.146997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.145106 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.147079 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.146466 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.147944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.147761 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.150113 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.150092 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.150591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.150566 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.207854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.207825 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.208262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.208221 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5m4z\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\") pod \"e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.267084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.267029 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:29.703221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.703189 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:51:29.704089 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:29.704065 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf9cc7_9c21_482d_be72_89afa7e05dd3.slice/crio-775eab5e8721ef3ce246e87bf109b6d3631881e989afbaffd597499d642176f7 WatchSource:0}: Error finding container 775eab5e8721ef3ce246e87bf109b6d3631881e989afbaffd597499d642176f7: Status 404 returned error can't find the container with id 775eab5e8721ef3ce246e87bf109b6d3631881e989afbaffd597499d642176f7 Oct 30 21:51:29.706938 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.706904 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:29.707017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.706980 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:29.707017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:29.707012 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:30.677242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:30.677204 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerStarted","Data":"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256"} Oct 30 21:51:30.677426 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:30.677291 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerStarted","Data":"775eab5e8721ef3ce246e87bf109b6d3631881e989afbaffd597499d642176f7"} Oct 30 21:51:30.851314 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:30.851261 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podStartSLOduration=2.851244932 podStartE2EDuration="2.851244932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:30.85010896 +0000 UTC m=+1715.524155499" watchObservedRunningTime="2026-10-30 21:51:30.851244932 +0000 UTC m=+1715.525291469" Oct 30 21:51:31.383631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.382463 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:31.420635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.400964 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:31.514247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.510075 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 21:51:31.520714 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.520687 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:31.577638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.575445 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-1a415f\"" Oct 30 21:51:31.601216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.600622 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-1a415f-dockercfg-m8w4m\"" Oct 30 21:51:31.721208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.721120 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 21:51:31.730094 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.730064 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:31.731696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.731662 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:51:31.770449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.764838 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:31.770449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.765020 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75g9\" (UniqueName: \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:31.770449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.766079 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 21:51:31.771795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.771756 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:31.849388 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.845415 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 21:51:31.936390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.928404 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg42h\" (UniqueName: \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:31.936390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.928482 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:31.936390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.928582 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:31.936390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.928706 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75g9\" (UniqueName: \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:31.966681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:31.960050 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:32.025770 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.025657 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75g9\" (UniqueName: \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\") pod \"payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:32.030661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.029937 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg42h\" (UniqueName: \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:32.030661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.030044 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:32.032856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.031473 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:32.108035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.107938 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg42h\" (UniqueName: \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\") pod \"payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:32.124435 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.124408 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:32.174419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.174377 4525 patch_prober.go:29] interesting pod/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:32.174642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.174453 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:32.247746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.247710 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:32.568061 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.568031 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 21:51:32.588139 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:32.588106 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491530a1_5be4_4429_a2ca_6299ce346b47.slice/crio-de80453c4a0a21b8073f669fe281f7795e4cb7e47e397ca1532eb9aa169419bf WatchSource:0}: Error finding container de80453c4a0a21b8073f669fe281f7795e4cb7e47e397ca1532eb9aa169419bf: Status 404 returned error can't find the container with id de80453c4a0a21b8073f669fe281f7795e4cb7e47e397ca1532eb9aa169419bf Oct 30 21:51:32.859564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.858269 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 21:51:32.864825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.864801 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:51:32.865895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.865869 4525 generic.go:356] "Generic (PLEG): container finished" podID="1b965a1d-b306-4d53-8c21-1912a802742c" containerID="b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93" exitCode=0 Oct 30 21:51:32.866138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.865955 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerDied","Data":"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93"} Oct 30 21:51:32.866287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.866269 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" event={"ID":"1b965a1d-b306-4d53-8c21-1912a802742c","Type":"ContainerDied","Data":"2334bda96e94533111df2c4320640363e7fc446300b3aaf0d90f2f359923f4f9"} Oct 30 21:51:32.866378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.866294 4525 scope.go:122] "RemoveContainer" containerID="b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93" Oct 30 21:51:32.868721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.868693 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" event={"ID":"491530a1-5be4-4429-a2ca-6299ce346b47","Type":"ContainerStarted","Data":"de80453c4a0a21b8073f669fe281f7795e4cb7e47e397ca1532eb9aa169419bf"} Oct 30 21:51:32.889130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.889099 4525 scope.go:122] "RemoveContainer" containerID="938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0" Oct 30 21:51:32.900063 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.900040 4525 scope.go:122] "RemoveContainer" containerID="b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93" Oct 30 21:51:32.900345 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:32.900326 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93\": container with ID starting with b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93 not found: ID does not exist" containerID="b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93" Oct 30 21:51:32.900417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.900353 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93"} err="failed to get container status \"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93\": rpc error: code = NotFound desc = could not find container \"b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93\": container with ID starting with b4bb14d6d258155c5a1b89626336892542b04bd876db7c50131f54dc75a44f93 not found: ID does not exist" Oct 30 21:51:32.900417 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.900371 4525 scope.go:122] "RemoveContainer" containerID="938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0" Oct 30 21:51:32.900658 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:32.900636 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0\": container with ID starting with 938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0 not found: ID does not exist" containerID="938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0" Oct 30 21:51:32.900712 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.900662 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0"} err="failed to get container status \"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0\": rpc error: code = NotFound desc = could not find container \"938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0\": container with ID starting with 938293e9c44e1ee23a03d3b71e046009bf710e05f2db8e8c4cbbc9dbecde18b0 not found: ID does not exist" Oct 30 21:51:32.947012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.946986 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947030 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947142 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947169 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947191 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947270 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947362 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947356 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947341 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:32.947425 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947418 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:32.947871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947421 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947523 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\") pod \"1b965a1d-b306-4d53-8c21-1912a802742c\" (UID: \"1b965a1d-b306-4d53-8c21-1912a802742c\") " Oct 30 21:51:32.947871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947812 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:32.947871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.947831 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:32.948589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.948567 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.948765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.948748 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b965a1d-b306-4d53-8c21-1912a802742c-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.948837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.948768 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.948837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.948782 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.948837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.948795 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:32.950164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.950143 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:32.950260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.950175 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "kube-api-access-fgxhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:32.950260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.950192 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:32.950353 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:32.950342 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo" pod "1b965a1d-b306-4d53-8c21-1912a802742c" (UID: "1b965a1d-b306-4d53-8c21-1912a802742c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:33.050623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.050579 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b965a1d-b306-4d53-8c21-1912a802742c-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.050623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.050623 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxhs\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-kube-api-access-fgxhs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.050836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.050635 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b965a1d-b306-4d53-8c21-1912a802742c-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.050836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.050645 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b965a1d-b306-4d53-8c21-1912a802742c-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.877185 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.877156 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2" Oct 30 21:51:33.879192 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.879154 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" event={"ID":"491530a1-5be4-4429-a2ca-6299ce346b47","Type":"ContainerStarted","Data":"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924"} Oct 30 21:51:33.879346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.879238 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:33.881175 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.881150 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" event={"ID":"93b4a926-8cc2-4297-a68b-510d6e507366","Type":"ContainerStarted","Data":"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346"} Oct 30 21:51:33.881310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.881196 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" event={"ID":"93b4a926-8cc2-4297-a68b-510d6e507366","Type":"ContainerStarted","Data":"8cb822e2616fa2f0018529fc33b9d59406404e017d4aec8121186c949ffbf770"} Oct 30 21:51:33.881310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:33.881297 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:34.046872 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.046810 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" podStartSLOduration=3.046792405 podStartE2EDuration="3.046792405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.045502139 +0000 UTC m=+1718.719548672" watchObservedRunningTime="2026-10-30 21:51:34.046792405 +0000 UTC m=+1718.720838943" Oct 30 21:51:34.165888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.165786 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" podStartSLOduration=3.165772273 podStartE2EDuration="3.165772273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.162213835 +0000 UTC m=+1718.836260371" watchObservedRunningTime="2026-10-30 21:51:34.165772273 +0000 UTC m=+1718.839818818" Oct 30 21:51:34.316629 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.315669 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:51:34.329638 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.322370 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-21084e54-gw-openshift-default-6cc68796d9-hnmx2"] Oct 30 21:51:34.460509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.460414 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:51:34.460723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:34.460694 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" containerID="cri-o://29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6" gracePeriod=30 Oct 30 21:51:35.397665 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:35.397595 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:51:35.474975 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.474946 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" path="/var/lib/kubelet/pods/1b965a1d-b306-4d53-8c21-1912a802742c/volumes" Oct 30 21:51:35.905593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.899296 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:51:35.915350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.914968 4525 cpu_manager.go:414] "containerMap: removing container" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:35.915350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.914997 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:35.915350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.915226 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:35.915350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.915243 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:35.938631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:35.937673 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.017633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.015855 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c92783-openshift-default-dockercfg-8zddj\"" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.035685 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7927h\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.035739 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.035830 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.035901 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.036006 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.036070 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.036266 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.036292 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.036341 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.065631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.041743 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148107 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148197 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148356 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148384 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148432 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148529 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7927h\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148560 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148655 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.148724 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.181699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.149542 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.182829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.182799 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.183242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.183223 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.183649 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.183627 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.225285 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.184816 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.298379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.295435 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.298379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.295971 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.298379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.296811 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7927h\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.305091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.305058 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\") pod \"e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.344361 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.341954 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:36.901879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.901850 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:51:36.902317 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:36.902292 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc858730_9758_465a_b392_ea601fcc791d.slice/crio-1faeaa3fa5022552fb94a0e23a07598c9d19292e4408386f0f53c42615e32833 WatchSource:0}: Error finding container 1faeaa3fa5022552fb94a0e23a07598c9d19292e4408386f0f53c42615e32833: Status 404 returned error can't find the container with id 1faeaa3fa5022552fb94a0e23a07598c9d19292e4408386f0f53c42615e32833 Oct 30 21:51:36.905229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.905195 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:36.905331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.905261 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:36.905331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.905291 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:51:36.923997 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:36.923973 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerStarted","Data":"1faeaa3fa5022552fb94a0e23a07598c9d19292e4408386f0f53c42615e32833"} Oct 30 21:51:37.933096 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:37.933061 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerStarted","Data":"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb"} Oct 30 21:51:38.348260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:38.348179 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:38.472321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:38.472279 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:38.615702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:38.614159 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podStartSLOduration=3.6141415 podStartE2EDuration="3.6141415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:38.1824993 +0000 UTC m=+1722.856545837" watchObservedRunningTime="2026-10-30 21:51:38.6141415 +0000 UTC m=+1723.288188038" Oct 30 21:51:39.253838 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.253702 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:39.256246 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.255912 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:51:39.759540 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.759461 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:51:39.816567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816529 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.816567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816571 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.816830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816660 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.816830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816743 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.816929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816862 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.816929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816902 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.817037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.816931 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.817037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817008 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:39.817141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817075 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.817141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817130 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\") pod \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\" (UID: \"a8d54bb1-ebb4-4a28-895f-67c22eb20b71\") " Oct 30 21:51:39.817251 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817169 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:39.817311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817291 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:39.817404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817375 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:39.817771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.817742 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:39.818072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.818040 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.818072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.818069 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.818220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.818085 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.818220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.818098 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.818220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.818112 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.819134 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.819108 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:39.819534 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.819508 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "kube-api-access-mssbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:39.819634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.819574 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:39.819634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.819582 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy" pod "a8d54bb1-ebb4-4a28-895f-67c22eb20b71" (UID: "a8d54bb1-ebb4-4a28-895f-67c22eb20b71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:39.939382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.939311 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.939382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.939346 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.939771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.939726 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mssbw\" (UniqueName: \"kubernetes.io/projected/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-kube-api-access-mssbw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:39.940037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:39.939923 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d54bb1-ebb4-4a28-895f-67c22eb20b71-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:40.262316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.262276 4525 generic.go:356] "Generic (PLEG): container finished" podID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerID="29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6" exitCode=0 Oct 30 21:51:40.262795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.262327 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" Oct 30 21:51:40.262795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.262368 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerDied","Data":"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6"} Oct 30 21:51:40.262795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.262431 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8" event={"ID":"a8d54bb1-ebb4-4a28-895f-67c22eb20b71","Type":"ContainerDied","Data":"74820522c11a78349ed50f7483998225f0cb981d838acf46ce769d49656e37d2"} Oct 30 21:51:40.262795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.262457 4525 scope.go:122] "RemoveContainer" containerID="29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6" Oct 30 21:51:40.279488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.279461 4525 scope.go:122] "RemoveContainer" containerID="e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa" Oct 30 21:51:40.290529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.290506 4525 scope.go:122] "RemoveContainer" containerID="29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6" Oct 30 21:51:40.290878 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:40.290853 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6\": container with ID starting with 29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6 not found: ID does not exist" containerID="29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6" Oct 30 21:51:40.290953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.290894 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6"} err="failed to get container status \"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6\": rpc error: code = NotFound desc = could not find container \"29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6\": container with ID starting with 29cbf8a46b231342966522da3ede309d029a0c079bb3c853e3c344eb98bdada6 not found: ID does not exist" Oct 30 21:51:40.290953 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.290920 4525 scope.go:122] "RemoveContainer" containerID="e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa" Oct 30 21:51:40.291230 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:40.291212 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa\": container with ID starting with e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa not found: ID does not exist" containerID="e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa" Oct 30 21:51:40.291274 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.291237 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa"} err="failed to get container status \"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa\": rpc error: code = NotFound desc = could not find container \"e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa\": container with ID starting with e459c4ee64ac68f3267cae40dcc15151b2593ad9600996565780d99bca97bcfa not found: ID does not exist" Oct 30 21:51:40.430530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.430496 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:51:40.480205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:40.480175 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fd496e14-openshift-default-6dd64fcfb5-q8mx8"] Oct 30 21:51:41.475506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:41.475474 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" path="/var/lib/kubelet/pods/a8d54bb1-ebb4-4a28-895f-67c22eb20b71/volumes" Oct 30 21:51:44.918632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:44.917945 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 21:51:44.924763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:44.919621 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 21:51:45.136637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.135903 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156762 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156794 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156814 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156822 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156907 4525 cpu_manager.go:414] "containerMap: removing container" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.156916 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b965a1d-b306-4d53-8c21-1912a802742c" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.157106 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.161963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.157122 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d54bb1-ebb4-4a28-895f-67c22eb20b71" containerName="istio-proxy" Oct 30 21:51:45.328690 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.324886 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:51:45.334620 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.334566 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.452628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.443223 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.452628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.443293 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scj5j\" (UniqueName: \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.452628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.446074 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c92783-dockercfg-pv9dx\"" Oct 30 21:51:45.452628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.447949 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c92783\"" Oct 30 21:51:45.598155 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.598071 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.599845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.599824 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scj5j\" (UniqueName: \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.600864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.599729 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.679288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.679252 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scj5j\" (UniqueName: \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\") pod \"payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:45.750206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:45.750174 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:46.212733 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:51:46.212707 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf48bde5_b0d7_4c9c_ac0f_38eb63380cdd.slice/crio-a1d1bd60b9de46e2c79ee520a544fdcb740f09041c99663194ff4db81bf3159d WatchSource:0}: Error finding container a1d1bd60b9de46e2c79ee520a544fdcb740f09041c99663194ff4db81bf3159d: Status 404 returned error can't find the container with id a1d1bd60b9de46e2c79ee520a544fdcb740f09041c99663194ff4db81bf3159d Oct 30 21:51:46.213153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:46.213118 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:51:46.389102 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:46.389059 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" event={"ID":"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd","Type":"ContainerStarted","Data":"7936ecc5765461e14c3ce400148dc5185eba37da946ce19a2d99f7fd3b892568"} Oct 30 21:51:46.389321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:46.389297 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:46.389415 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:46.389387 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" event={"ID":"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd","Type":"ContainerStarted","Data":"a1d1bd60b9de46e2c79ee520a544fdcb740f09041c99663194ff4db81bf3159d"} Oct 30 21:51:46.509494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:46.509398 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" podStartSLOduration=1.5093833490000002 podStartE2EDuration="1.509383349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:46.506670852 +0000 UTC m=+1731.180717389" watchObservedRunningTime="2026-10-30 21:51:46.509383349 +0000 UTC m=+1731.183429884" Oct 30 21:51:53.751913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.751878 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:51:53.752287 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.752101 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" containerID="cri-o://7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b" gracePeriod=30 Oct 30 21:51:53.801936 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.801902 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:51:53.802178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.802155 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" containerID="cri-o://b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44" gracePeriod=30 Oct 30 21:51:53.805726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.805698 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:51:53.805920 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.805899 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" containerID="cri-o://ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689" gracePeriod=30 Oct 30 21:51:53.853143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.853113 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:51:53.853403 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.853364 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" containerID="cri-o://0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd" gracePeriod=30 Oct 30 21:51:53.857824 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.857794 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:51:53.858064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:53.858037 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" containerID="cri-o://8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6" gracePeriod=30 Oct 30 21:51:54.844174 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:54.844139 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:51:56.099542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:56.099513 4525 scope.go:122] "RemoveContainer" containerID="c8e8a64819f06bdef4b34ee5cb1e534a44a45e3df3cbf75d314b8d1e14302b34" Oct 30 21:51:57.398876 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:57.398843 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:51:59.363718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.363687 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:51:59.388748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.388722 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:51:59.424569 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424533 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.424764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424588 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.424764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424650 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.424764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424709 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.424764 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424759 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424788 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424854 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424912 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424958 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425025 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.424994 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425030 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425086 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425131 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425176 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425265 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425230 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425310 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.425476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425338 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\") pod \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\" (UID: \"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0\") " Oct 30 21:51:59.425476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.425379 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\") pod \"32ce90b0-b056-49bb-8e08-85af6bb2a480\" (UID: \"32ce90b0-b056-49bb-8e08-85af6bb2a480\") " Oct 30 21:51:59.429523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.428245 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.429523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.428454 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.429523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.429193 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.439761 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.439974 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.440249 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.440713 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.441294 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.441770 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.462677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.441955 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.477069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.477021 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.486870 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.486827 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "kube-api-access-j5pft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.487053 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.486966 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.487230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.487131 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.490661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.489801 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.490661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.490219 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.490661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.490383 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token" pod "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" (UID: "4772b666-2d8b-4fc7-b00d-4f79bb9a19b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.495661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.494932 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx" pod "32ce90b0-b056-49bb-8e08-85af6bb2a480" (UID: "32ce90b0-b056-49bb-8e08-85af6bb2a480"). InnerVolumeSpecName "kube-api-access-vbmbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.515644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.515580 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:51:59.519481 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.519455 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:51:59.524316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.524287 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:51:59.526774 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526751 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.526882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526796 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.526882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526827 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526882 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526940 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.526971 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527083 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527153 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527121 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527245 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527174 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527245 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527206 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527262 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527303 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527333 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527363 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527392 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527429 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527481 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527549 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527630 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527682 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527717 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527768 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\") pod \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\" (UID: \"2ffcd584-3c93-4ffa-b948-6abeb9be9733\") " Oct 30 21:51:59.527959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527844 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.527959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527883 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.527959 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.527918 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.528160 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528002 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\") pod \"7548d7d5-cd89-47d3-bab9-04ea46828673\" (UID: \"7548d7d5-cd89-47d3-bab9-04ea46828673\") " Oct 30 21:51:59.528160 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528043 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\") pod \"419137d2-3574-453c-b867-90064a172154\" (UID: \"419137d2-3574-453c-b867-90064a172154\") " Oct 30 21:51:59.528729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528709 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528733 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528747 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528760 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528774 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pft\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-kube-api-access-j5pft\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528788 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528802 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528814 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.528833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528827 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ce90b0-b056-49bb-8e08-85af6bb2a480-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528840 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528856 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528870 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmbx\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-kube-api-access-vbmbx\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528885 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528900 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528914 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528928 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528941 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ce90b0-b056-49bb-8e08-85af6bb2a480-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.528956 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.529387 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.529394 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.529403 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.529600 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.540164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.529663 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.556159 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.555024 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.559169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.558050 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.559169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.558307 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.559169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.558549 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.559169 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.558874 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.561294 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.559868 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:59.561294 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.560350 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.561294 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.560511 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.562932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.562232 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.562932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.562452 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.562932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.562676 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.574000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.573956 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.576202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576113 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "kube-api-access-4tsc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.576202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576157 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "kube-api-access-hs9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.576202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576181 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "kube-api-access-slg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.576571 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576547 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.576668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576646 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.576743 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576727 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo" pod "2ffcd584-3c93-4ffa-b948-6abeb9be9733" (UID: "2ffcd584-3c93-4ffa-b948-6abeb9be9733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.576903 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.576882 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:59.577566 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.577532 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:59.577919 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.577891 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token" pod "419137d2-3574-453c-b867-90064a172154" (UID: "419137d2-3574-453c-b867-90064a172154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582077 4525 generic.go:356] "Generic (PLEG): container finished" podID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerID="b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44" exitCode=0 Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582137 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token" pod "7548d7d5-cd89-47d3-bab9-04ea46828673" (UID: "7548d7d5-cd89-47d3-bab9-04ea46828673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582185 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" event={"ID":"2ffcd584-3c93-4ffa-b948-6abeb9be9733","Type":"ContainerDied","Data":"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44"} Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582224 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" event={"ID":"2ffcd584-3c93-4ffa-b948-6abeb9be9733","Type":"ContainerDied","Data":"67a84c338819bc8df1982bc67401c1702d42cb69354dc067fa187824f662d3fe"} Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582245 4525 scope.go:122] "RemoveContainer" containerID="b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44" Oct 30 21:51:59.582636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.582363 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh" Oct 30 21:51:59.591270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.588541 4525 generic.go:356] "Generic (PLEG): container finished" podID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerID="7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b" exitCode=0 Oct 30 21:51:59.591270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.588643 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" event={"ID":"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0","Type":"ContainerDied","Data":"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b"} Oct 30 21:51:59.591270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.588681 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" event={"ID":"4772b666-2d8b-4fc7-b00d-4f79bb9a19b0","Type":"ContainerDied","Data":"5c09fde40ad732e2ec4aaff9a0fb6cee8cc0a9fe938864f67de8d98d43ad783a"} Oct 30 21:51:59.591270 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.588742 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59" Oct 30 21:51:59.594373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.594332 4525 generic.go:356] "Generic (PLEG): container finished" podID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerID="0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd" exitCode=0 Oct 30 21:51:59.594694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.594673 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" event={"ID":"7548d7d5-cd89-47d3-bab9-04ea46828673","Type":"ContainerDied","Data":"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd"} Oct 30 21:51:59.594887 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.594873 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" event={"ID":"7548d7d5-cd89-47d3-bab9-04ea46828673","Type":"ContainerDied","Data":"8bf27fe7aea8241f1466a3dcd87dd04bbd3ddb1373d099088cbebf80a4046575"} Oct 30 21:51:59.595100 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.595087 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx" Oct 30 21:51:59.602718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.602690 4525 generic.go:356] "Generic (PLEG): container finished" podID="419137d2-3574-453c-b867-90064a172154" containerID="8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6" exitCode=0 Oct 30 21:51:59.602941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.602766 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerDied","Data":"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6"} Oct 30 21:51:59.603027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.602990 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" event={"ID":"419137d2-3574-453c-b867-90064a172154","Type":"ContainerDied","Data":"29085e9f19ab4a67e1592a37615971cf5e560ed1b9d7b2dfa0dbe87b7524ae9b"} Oct 30 21:51:59.603027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.602804 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7" Oct 30 21:51:59.605589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.605569 4525 scope.go:122] "RemoveContainer" containerID="b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44" Oct 30 21:51:59.606128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.606102 4525 generic.go:356] "Generic (PLEG): container finished" podID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerID="ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689" exitCode=0 Oct 30 21:51:59.606261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.606156 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" event={"ID":"32ce90b0-b056-49bb-8e08-85af6bb2a480","Type":"ContainerDied","Data":"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689"} Oct 30 21:51:59.606261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.606197 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" event={"ID":"32ce90b0-b056-49bb-8e08-85af6bb2a480","Type":"ContainerDied","Data":"d4aab0b49ac95dd3d121991a4a93e6fe14e849f0a6ceaef7d8b27927776bf945"} Oct 30 21:51:59.606393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.606268 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9" Oct 30 21:51:59.610331 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.610103 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44\": container with ID starting with b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44 not found: ID does not exist" containerID="b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44" Oct 30 21:51:59.610331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.610143 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44"} err="failed to get container status \"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44\": rpc error: code = NotFound desc = could not find container \"b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44\": container with ID starting with b8dc5b2de8856664bbf6034af5419b99a23aacce17304c0df5b7ff733760da44 not found: ID does not exist" Oct 30 21:51:59.610331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.610171 4525 scope.go:122] "RemoveContainer" containerID="7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b" Oct 30 21:51:59.625275 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.625246 4525 scope.go:122] "RemoveContainer" containerID="7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b" Oct 30 21:51:59.625642 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.625583 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b\": container with ID starting with 7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b not found: ID does not exist" containerID="7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b" Oct 30 21:51:59.625733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.625632 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b"} err="failed to get container status \"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b\": rpc error: code = NotFound desc = could not find container \"7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b\": container with ID starting with 7c9307e8c66ea68aca9ddb9730f1e40fa005ce09e23c0f77a655019e113c711b not found: ID does not exist" Oct 30 21:51:59.625733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.625654 4525 scope.go:122] "RemoveContainer" containerID="0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630670 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419137d2-3574-453c-b867-90064a172154-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630698 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tsc6\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-kube-api-access-4tsc6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630714 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630726 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630740 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630754 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630767 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630781 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630794 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630807 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630819 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630833 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630846 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630857 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630871 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630887 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9g7\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-kube-api-access-hs9g7\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630902 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630915 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630928 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffcd584-3c93-4ffa-b948-6abeb9be9733-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630943 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7548d7d5-cd89-47d3-bab9-04ea46828673-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630956 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419137d2-3574-453c-b867-90064a172154-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630968 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419137d2-3574-453c-b867-90064a172154-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630981 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slg2l\" (UniqueName: \"kubernetes.io/projected/7548d7d5-cd89-47d3-bab9-04ea46828673-kube-api-access-slg2l\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.630996 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.631012 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419137d2-3574-453c-b867-90064a172154-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.631024 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7548d7d5-cd89-47d3-bab9-04ea46828673-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.640994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.631037 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffcd584-3c93-4ffa-b948-6abeb9be9733-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:59.664751 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.664675 4525 scope.go:122] "RemoveContainer" containerID="0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd" Oct 30 21:51:59.665811 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.665755 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd\": container with ID starting with 0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd not found: ID does not exist" containerID="0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd" Oct 30 21:51:59.666084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.666029 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd"} err="failed to get container status \"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd\": rpc error: code = NotFound desc = could not find container \"0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd\": container with ID starting with 0b4b3169d3cb603d65bf8ff354da5d71ced6e59ebcae5fe9e69e1814eccb2ecd not found: ID does not exist" Oct 30 21:51:59.673965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.673937 4525 scope.go:122] "RemoveContainer" containerID="8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6" Oct 30 21:51:59.687565 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.687201 4525 scope.go:122] "RemoveContainer" containerID="f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a" Oct 30 21:51:59.699725 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.699697 4525 scope.go:122] "RemoveContainer" containerID="8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6" Oct 30 21:51:59.700165 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.700135 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6\": container with ID starting with 8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6 not found: ID does not exist" containerID="8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6" Oct 30 21:51:59.700252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.700179 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6"} err="failed to get container status \"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6\": rpc error: code = NotFound desc = could not find container \"8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6\": container with ID starting with 8acc1fb297df9be1b56bb4b08cbe22322a959b99cbbd7aad368ac5820f7a68c6 not found: ID does not exist" Oct 30 21:51:59.700252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.700204 4525 scope.go:122] "RemoveContainer" containerID="f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a" Oct 30 21:51:59.700961 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.700586 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a\": container with ID starting with f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a not found: ID does not exist" containerID="f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a" Oct 30 21:51:59.700961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.700647 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a"} err="failed to get container status \"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a\": rpc error: code = NotFound desc = could not find container \"f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a\": container with ID starting with f75172967250b64c95357cb1d2c1afbe52417d81a137325bf6e2ac0cf0a7455a not found: ID does not exist" Oct 30 21:51:59.700961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.700669 4525 scope.go:122] "RemoveContainer" containerID="ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689" Oct 30 21:51:59.713929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.713910 4525 scope.go:122] "RemoveContainer" containerID="ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689" Oct 30 21:51:59.714313 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:51:59.714291 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689\": container with ID starting with ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689 not found: ID does not exist" containerID="ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689" Oct 30 21:51:59.714378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:51:59.714322 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689"} err="failed to get container status \"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689\": rpc error: code = NotFound desc = could not find container \"ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689\": container with ID starting with ea819bb1208be6453f96cec80b2ddd34f5db8165b4d5f76edfd5c6521e7a6689 not found: ID does not exist" Oct 30 21:52:00.315254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.315220 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:52:00.321115 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.321087 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-966nx"] Oct 30 21:52:00.459596 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.459552 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:52:00.463240 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.463211 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-wmz59"] Oct 30 21:52:00.569789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.569671 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:52:00.574091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.574061 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-ph5vh"] Oct 30 21:52:00.684211 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.684180 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:52:00.687586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.687567 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-ph9r9"] Oct 30 21:52:00.816625 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.816582 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:52:00.821482 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:00.821420 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c7d477-openshift-default-776595695-qkgr7"] Oct 30 21:52:01.474643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:01.474591 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" path="/var/lib/kubelet/pods/2ffcd584-3c93-4ffa-b948-6abeb9be9733/volumes" Oct 30 21:52:01.475135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:01.475120 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" path="/var/lib/kubelet/pods/32ce90b0-b056-49bb-8e08-85af6bb2a480/volumes" Oct 30 21:52:01.475570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:01.475556 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419137d2-3574-453c-b867-90064a172154" path="/var/lib/kubelet/pods/419137d2-3574-453c-b867-90064a172154/volumes" Oct 30 21:52:01.476320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:01.476306 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" path="/var/lib/kubelet/pods/4772b666-2d8b-4fc7-b00d-4f79bb9a19b0/volumes" Oct 30 21:52:01.476766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:01.476753 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" path="/var/lib/kubelet/pods/7548d7d5-cd89-47d3-bab9-04ea46828673/volumes" Oct 30 21:52:05.164730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:05.164697 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:52:05.165163 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:05.164947 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" containerID="cri-o://84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70" gracePeriod=30 Oct 30 21:52:05.402330 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:05.402292 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:52:06.670071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.670039 4525 generic.go:356] "Generic (PLEG): container finished" podID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerID="8b8abf001bcacf4c0a475268b3abd4cca1e04714fd599b9bf077065c60d6b573" exitCode=0 Oct 30 21:52:06.670515 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.670101 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerDied","Data":"8b8abf001bcacf4c0a475268b3abd4cca1e04714fd599b9bf077065c60d6b573"} Oct 30 21:52:06.670561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.670524 4525 scope.go:122] "RemoveContainer" containerID="8b8abf001bcacf4c0a475268b3abd4cca1e04714fd599b9bf077065c60d6b573" Oct 30 21:52:06.672363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.672324 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:06.672481 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.672410 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:06.672481 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:06.672444 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:07.678427 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:07.678389 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerStarted","Data":"816ab21c5773e6064ee0953ca8ec37b81494f4dd51ec7d0aeb56d990ed7b9740"} Oct 30 21:52:08.037822 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:08.037719 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:52:08.601647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:08.601600 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:52:08.602805 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:08.602770 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:08.602948 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:08.602835 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:52:09.601535 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:09.601499 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:09.602036 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:09.601564 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:52:09.856725 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:09.856633 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:52:09.856725 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:09.856713 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:52:10.490620 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.490589 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:52:10.594582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594552 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.594797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594596 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.594797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594653 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.594797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594692 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.594797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594737 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.594797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594794 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.595087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594830 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.595087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594865 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.595087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.594964 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\") pod \"9e981faa-c724-46a7-bc34-0882165d274f\" (UID: \"9e981faa-c724-46a7-bc34-0882165d274f\") " Oct 30 21:52:10.595087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.595026 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.595087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.595068 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:10.595405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.595381 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.595593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.595566 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.596293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.596262 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.596293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.596286 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e981faa-c724-46a7-bc34-0882165d274f-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.596469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.596300 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.596469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.596316 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.596469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.596394 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.597597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.597576 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:10.597733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.597705 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:10.597968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.597944 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:52:10.598236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.598222 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t" pod "9e981faa-c724-46a7-bc34-0882165d274f" (UID: "9e981faa-c724-46a7-bc34-0882165d274f"). InnerVolumeSpecName "kube-api-access-bwj5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:10.601549 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.601528 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:10.601911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.601573 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:52:10.697511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.697479 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.697511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.697507 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.697511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.697520 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e981faa-c724-46a7-bc34-0882165d274f-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.697857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.697535 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e981faa-c724-46a7-bc34-0882165d274f-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.697857 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.697546 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwj5t\" (UniqueName: \"kubernetes.io/projected/9e981faa-c724-46a7-bc34-0882165d274f-kube-api-access-bwj5t\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:10.704895 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.704865 4525 generic.go:356] "Generic (PLEG): container finished" podID="9e981faa-c724-46a7-bc34-0882165d274f" containerID="84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70" exitCode=0 Oct 30 21:52:10.705070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.704965 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" event={"ID":"9e981faa-c724-46a7-bc34-0882165d274f","Type":"ContainerDied","Data":"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70"} Oct 30 21:52:10.705070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.705007 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" event={"ID":"9e981faa-c724-46a7-bc34-0882165d274f","Type":"ContainerDied","Data":"1634ea737db6dac1194410bf89d2461ac4679c236b12702948924352288ce839"} Oct 30 21:52:10.705070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.705027 4525 scope.go:122] "RemoveContainer" containerID="84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70" Oct 30 21:52:10.705250 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.705148 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm" Oct 30 21:52:10.718975 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.718957 4525 scope.go:122] "RemoveContainer" containerID="84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70" Oct 30 21:52:10.719238 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:10.719219 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70\": container with ID starting with 84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70 not found: ID does not exist" containerID="84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70" Oct 30 21:52:10.719332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.719251 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70"} err="failed to get container status \"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70\": rpc error: code = NotFound desc = could not find container \"84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70\": container with ID starting with 84251301fb1202917d6e6d16c91641cf325699775ab88ccc59dfe75bb2a51b70 not found: ID does not exist" Oct 30 21:52:10.875109 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.875076 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:52:10.878421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:10.878391 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-fnbcm"] Oct 30 21:52:11.475061 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:11.475029 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e981faa-c724-46a7-bc34-0882165d274f" path="/var/lib/kubelet/pods/9e981faa-c724-46a7-bc34-0882165d274f/volumes" Oct 30 21:52:11.601066 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:11.601031 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:11.601273 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:11.601089 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:52:12.601931 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:12.601890 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:52:12.602447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:12.601951 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 21:52:13.606389 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:13.606296 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:52:13.607332 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:13.607310 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 21:52:31.005813 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.005780 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 21:52:31.006469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006445 4525 cpu_manager.go:414] "containerMap: removing container" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" Oct 30 21:52:31.006469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006463 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" Oct 30 21:52:31.006469 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006476 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006481 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006489 4525 cpu_manager.go:414] "containerMap: removing container" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006494 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006501 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006506 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006520 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006524 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006534 4525 cpu_manager.go:414] "containerMap: removing container" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006538 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006556 4525 cpu_manager.go:414] "containerMap: removing container" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006561 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006693 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e981faa-c724-46a7-bc34-0882165d274f" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006704 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7548d7d5-cd89-47d3-bab9-04ea46828673" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006711 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006716 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffcd584-3c93-4ffa-b948-6abeb9be9733" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006724 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4772b666-2d8b-4fc7-b00d-4f79bb9a19b0" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006732 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="419137d2-3574-453c-b867-90064a172154" containerName="istio-proxy" Oct 30 21:52:31.006756 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.006741 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ce90b0-b056-49bb-8e08-85af6bb2a480" containerName="istio-proxy" Oct 30 21:52:31.010665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.010641 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.063897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.063869 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-1a415f-serving-cert\"" Oct 30 21:52:31.099683 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.099641 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq7xc\" (UniqueName: \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.099915 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.099893 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.100037 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.100019 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.100164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.100142 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.117787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.117753 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 21:52:31.201538 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.201509 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.201747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.201586 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.201747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.201671 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.201830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.201794 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq7xc\" (UniqueName: \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.202138 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:31.202108 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-serving-cert: secret "maas-api-e2e-shared-b-w0-1a415f-serving-cert" not found Oct 30 21:52:31.202217 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:31.202208 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls podName:25ad9478-4720-47a3-a9b6-3df6938e5315 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:31.702187444 +0000 UTC m=+1776.376233973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls") pod "maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" (UID: "25ad9478-4720-47a3-a9b6-3df6938e5315") : secret "maas-api-e2e-shared-b-w0-1a415f-serving-cert" not found Oct 30 21:52:31.202290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.202229 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.204557 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.204533 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.289681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.288886 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq7xc\" (UniqueName: \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.719124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.719084 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.721862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.721833 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:31.926538 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:31.926503 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:32.355039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:32.354947 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 21:52:32.360325 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:52:32.360294 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ad9478_4720_47a3_a9b6_3df6938e5315.slice/crio-d6140c8ce9aee51860016592fb48f3ddabbe3003aced1098171e98f75df159a1 WatchSource:0}: Error finding container d6140c8ce9aee51860016592fb48f3ddabbe3003aced1098171e98f75df159a1: Status 404 returned error can't find the container with id d6140c8ce9aee51860016592fb48f3ddabbe3003aced1098171e98f75df159a1 Oct 30 21:52:32.901760 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:32.901722 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" event={"ID":"25ad9478-4720-47a3-a9b6-3df6938e5315","Type":"ContainerStarted","Data":"d6140c8ce9aee51860016592fb48f3ddabbe3003aced1098171e98f75df159a1"} Oct 30 21:52:33.910340 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:33.910289 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" event={"ID":"25ad9478-4720-47a3-a9b6-3df6938e5315","Type":"ContainerStarted","Data":"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb"} Oct 30 21:52:33.910803 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:33.910353 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:34.031640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.031566 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" podStartSLOduration=3.191867114 podStartE2EDuration="4.031550045s" totalImagesPullingTime="839.682931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:34.029133635 +0000 UTC m=+1778.703180178" watchObservedRunningTime="2026-10-30 21:52:34.031550045 +0000 UTC m=+1778.705596593" Oct 30 21:52:34.762646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.762588 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 21:52:34.768152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.768128 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.825932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.825902 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 21:52:34.869498 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.869463 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-71a3d788\"/\"openshift-service-ca.crt\"" Oct 30 21:52:34.869498 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.869463 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-71a3d788\"/\"kube-root-ca.crt\"" Oct 30 21:52:34.869723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.869467 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-71a3d788\"/\"default-dockercfg-nh54x\"" Oct 30 21:52:34.869723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.869538 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-71a3d788\"/\"e2e-lifecycle-model-71a3d788-kserve-self-signed-certs\"" Oct 30 21:52:34.894768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.894737 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnnq\" (UniqueName: \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.894933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.894795 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.894975 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.894930 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.894975 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.894956 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.895131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.895112 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:34.895166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:34.895150 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000447 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnnq\" (UniqueName: \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000533 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000646 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000673 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000778 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.000998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.000804 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.001696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.001668 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.002027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.001993 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.002157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.002112 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.004290 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.004269 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.004800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.004779 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.062511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.062434 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnnq\" (UniqueName: \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\") pod \"e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.082315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.082275 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:35.408336 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:35.408295 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:52:35.493093 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:52:35.493065 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6de23ea9_9274_4d0c_932e_6858f37d03cf.slice/crio-4e902f08c94a3f592c9d38829154bd49767a8b63e41cc93a20fcd59a05361b56 WatchSource:0}: Error finding container 4e902f08c94a3f592c9d38829154bd49767a8b63e41cc93a20fcd59a05361b56: Status 404 returned error can't find the container with id 4e902f08c94a3f592c9d38829154bd49767a8b63e41cc93a20fcd59a05361b56 Oct 30 21:52:35.495635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.495584 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 21:52:35.957499 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.957457 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" event={"ID":"6de23ea9-9274-4d0c-932e-6858f37d03cf","Type":"ContainerStarted","Data":"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519"} Oct 30 21:52:35.957732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:35.957517 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" event={"ID":"6de23ea9-9274-4d0c-932e-6858f37d03cf","Type":"ContainerStarted","Data":"4e902f08c94a3f592c9d38829154bd49767a8b63e41cc93a20fcd59a05361b56"} Oct 30 21:52:36.077878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:36.077819 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" podStartSLOduration=2.077801891 podStartE2EDuration="2.077801891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:36.074343374 +0000 UTC m=+1780.748389910" watchObservedRunningTime="2026-10-30 21:52:36.077801891 +0000 UTC m=+1780.751848426" Oct 30 21:52:36.083366 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:36.083327 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:38.982131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.982098 4525 generic.go:356] "Generic (PLEG): container finished" podID="e7554552-a966-45ce-a189-7abd23f315db" containerID="8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece" exitCode=0 Oct 30 21:52:38.982671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.982156 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerDied","Data":"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece"} Oct 30 21:52:38.982671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.982594 4525 scope.go:122] "RemoveContainer" containerID="8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece" Oct 30 21:52:38.984666 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.984596 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:38.984781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.984724 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:38.984781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:38.984771 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:39.959084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:39.959054 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 21:52:39.991965 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:39.991930 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerStarted","Data":"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441"} Oct 30 21:52:40.583176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:40.583142 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:52:40.584260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:40.584229 4525 patch_prober.go:29] interesting pod/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:52:40.584397 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:40.584290 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:52:41.457566 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.457523 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:52:41.462387 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.462363 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.511205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.511173 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:52:41.556859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.530174 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.556859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.530433 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hq2\" (UniqueName: \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.556859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.530571 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.556859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.530715 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.560632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.559169 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d6e0be-serving-cert\"" Oct 30 21:52:41.583540 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.583504 4525 patch_prober.go:29] interesting pod/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:52:41.583776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.583571 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.643026 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.643129 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.643204 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.643381 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hq2\" (UniqueName: \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:41.643815 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-serving-cert: secret "maas-api-e2e-shared-a-w4-d6e0be-serving-cert" not found Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:41.643889 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls podName:3be7072c-9090-4137-a303-45d5f2e3191d nodeName:}" failed. No retries permitted until 2026-10-30 21:52:42.143866736 +0000 UTC m=+1786.817913254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls") pod "maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" (UID: "3be7072c-9090-4137-a303-45d5f2e3191d") : secret "maas-api-e2e-shared-a-w4-d6e0be-serving-cert" not found Oct 30 21:52:41.653631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.644596 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.660759 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.660729 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:41.743080 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:41.743009 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hq2\" (UniqueName: \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:42.170562 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.170528 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:42.173252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.173224 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:42.377149 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.377103 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:42.582956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.582923 4525 patch_prober.go:29] interesting pod/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:52:42.583937 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.582969 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:52:42.781651 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:52:42.781584 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be7072c_9090_4137_a303_45d5f2e3191d.slice/crio-ffd85d77f5673837faf9d3fb4d14d5e0193c79452c5f4eccfaabad63e22ea0b2 WatchSource:0}: Error finding container ffd85d77f5673837faf9d3fb4d14d5e0193c79452c5f4eccfaabad63e22ea0b2: Status 404 returned error can't find the container with id ffd85d77f5673837faf9d3fb4d14d5e0193c79452c5f4eccfaabad63e22ea0b2 Oct 30 21:52:42.784596 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.784572 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:52:42.852745 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:42.852710 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:52:43.026496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:43.026413 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" event={"ID":"3be7072c-9090-4137-a303-45d5f2e3191d","Type":"ContainerStarted","Data":"ffd85d77f5673837faf9d3fb4d14d5e0193c79452c5f4eccfaabad63e22ea0b2"} Oct 30 21:52:43.582998 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:43.582967 4525 patch_prober.go:29] interesting pod/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:52:43.583402 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:43.583031 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:52:44.059384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:44.059343 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" event={"ID":"3be7072c-9090-4137-a303-45d5f2e3191d","Type":"ContainerStarted","Data":"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400"} Oct 30 21:52:44.059574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:44.059462 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:44.177399 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:44.177237 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" podStartSLOduration=2.184946445 podStartE2EDuration="3.177217686s" totalImagesPullingTime="992.271241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:44.177131275 +0000 UTC m=+1788.851177811" watchObservedRunningTime="2026-10-30 21:52:44.177217686 +0000 UTC m=+1788.851264223" Oct 30 21:52:44.583486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:44.583391 4525 patch_prober.go:29] interesting pod/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:52:44.583486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:44.583449 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:52:45.083030 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:45.082995 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:45.096596 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:45.096565 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:45.589342 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:45.589312 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:52:45.590575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:45.590556 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:52:46.086783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:46.086755 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 21:52:50.073448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:50.073416 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:52:51.596641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.596585 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:52:51.601167 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.601143 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.664540 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.664497 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4fpr\" (UniqueName: \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.664783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.664581 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.664783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.664708 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.664783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.664761 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.665062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.665035 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-c92783-serving-cert\"" Oct 30 21:52:51.721658 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.721619 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:52:51.766347 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.766317 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.766522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.766425 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.766522 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:51.766436 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-serving-cert: secret "maas-api-e2e-shared-a-w1-c92783-serving-cert" not found Oct 30 21:52:51.766522 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.766485 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.766663 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:52:51.766566 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls podName:05cffc26-74ce-4e79-93ad-8ba262b09e6f nodeName:}" failed. No retries permitted until 2026-10-30 21:52:52.266541018 +0000 UTC m=+1796.940587532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls") pod "maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" (UID: "05cffc26-74ce-4e79-93ad-8ba262b09e6f") : secret "maas-api-e2e-shared-a-w1-c92783-serving-cert" not found Oct 30 21:52:51.766932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.766901 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4fpr\" (UniqueName: \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.767173 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.767150 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.769168 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.769146 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:51.829189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:51.829151 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4fpr\" (UniqueName: \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:52.300884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:52.300844 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:52.303502 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:52.303476 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:52.532632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:52.524787 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:52.765901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:52.765862 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:52:52.996444 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:52:52.996416 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cffc26_74ce_4e79_93ad_8ba262b09e6f.slice/crio-84105c47cd0e5f0593ddf6ea171b4d7b004143561f64cc0421e18592b6abdf3b WatchSource:0}: Error finding container 84105c47cd0e5f0593ddf6ea171b4d7b004143561f64cc0421e18592b6abdf3b: Status 404 returned error can't find the container with id 84105c47cd0e5f0593ddf6ea171b4d7b004143561f64cc0421e18592b6abdf3b Oct 30 21:52:52.996954 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:52.996933 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:52:53.156900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:53.156866 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" event={"ID":"05cffc26-74ce-4e79-93ad-8ba262b09e6f","Type":"ContainerStarted","Data":"84105c47cd0e5f0593ddf6ea171b4d7b004143561f64cc0421e18592b6abdf3b"} Oct 30 21:52:54.233640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:54.231982 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" event={"ID":"05cffc26-74ce-4e79-93ad-8ba262b09e6f","Type":"ContainerStarted","Data":"770cf69ae21e8a95b34eb59c5a0c4250511fccbac1b75dcf88576a63203ea095"} Oct 30 21:52:54.233640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:54.233236 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:52:54.362842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:52:54.362786 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" podStartSLOduration=2.501530831 podStartE2EDuration="3.362767597s" totalImagesPullingTime="861.236766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:54.359019319 +0000 UTC m=+1799.033065856" watchObservedRunningTime="2026-10-30 21:52:54.362767597 +0000 UTC m=+1799.036814133" Oct 30 21:53:01.269048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:01.269017 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:53:04.323669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.323598 4525 generic.go:356] "Generic (PLEG): container finished" podID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerID="f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0" exitCode=0 Oct 30 21:53:04.324108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.323674 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerDied","Data":"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0"} Oct 30 21:53:04.324157 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.324141 4525 scope.go:122] "RemoveContainer" containerID="f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0" Oct 30 21:53:04.325941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.325904 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:04.326064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.325995 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:04.326064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:04.326044 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:05.357437 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:05.357398 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerStarted","Data":"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a"} Oct 30 21:53:05.398670 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:53:05.398595 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:53:06.275471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.275428 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:53:06.294684 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.293834 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:53:06.294684 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.293906 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:53:06.600787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.600696 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:53:06.612981 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.612953 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.666106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.666069 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:53:06.670401 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.670361 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-79e10a\"" Oct 30 21:53:06.671943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.671890 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-79e10a-dockercfg-wdtlh\"" Oct 30 21:53:06.698187 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.698147 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.698619 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.698572 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxs7b\" (UniqueName: \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.799958 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.799920 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxs7b\" (UniqueName: \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.800149 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.800020 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.800787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.800766 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.861707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.861584 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxs7b\" (UniqueName: \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\") pod \"payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:06.983402 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:06.983359 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:07.282635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:07.280372 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:53:07.282635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:07.280478 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:53:07.479682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:07.472582 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:53:08.269570 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.269533 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:53:08.270099 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.269587 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:53:08.493793 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.493760 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" event={"ID":"dd2164db-f8c6-4d60-8089-55394bb918ea","Type":"ContainerStarted","Data":"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2"} Oct 30 21:53:08.494007 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.493814 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:08.494007 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.493840 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" event={"ID":"dd2164db-f8c6-4d60-8089-55394bb918ea","Type":"ContainerStarted","Data":"91e2154971d62114b7105613973521142cbaf5327e654547ebb5ab25747889c1"} Oct 30 21:53:08.616874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:08.616815 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" podStartSLOduration=2.616796589 podStartE2EDuration="2.616796589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:08.612779421 +0000 UTC m=+1813.286825958" watchObservedRunningTime="2026-10-30 21:53:08.616796589 +0000 UTC m=+1813.290843126" Oct 30 21:53:09.270081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:09.270048 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:53:09.270524 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:09.270103 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:53:09.386921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:09.386886 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:53:10.270069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:10.270033 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:53:10.270252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:10.270087 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 21:53:11.311078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:11.311020 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:53:11.312791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:11.312765 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 21:53:19.504939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:19.504910 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:53:31.006195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:31.006163 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:53:35.408035 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:53:35.407997 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:53:40.296797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:40.296751 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:53:40.297350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:40.296836 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.229868 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.229831 4525 generic.go:356] "Generic (PLEG): container finished" podID="fc858730-9758-465a-b392-ea601fcc791d" containerID="0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb" exitCode=0 Oct 30 21:53:42.230282 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.229904 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerDied","Data":"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb"} Oct 30 21:53:42.230378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.230363 4525 scope.go:122] "RemoveContainer" containerID="0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb" Oct 30 21:53:42.232132 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.232098 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:42.232248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.232185 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:42.232312 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.232255 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:42.649407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.649362 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:53:42.655385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.655354 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:42.712623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.712578 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-bd3c59-dockercfg-nl65v\"" Oct 30 21:53:42.716722 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.716691 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:53:42.947687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.946989 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bd3c59\"" Oct 30 21:53:42.978027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.977927 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5nn5\" (UniqueName: \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:42.978591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:42.978561 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.080754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.080711 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.081861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.080790 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5nn5\" (UniqueName: \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.082214 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.082187 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.143552 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.143516 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5nn5\" (UniqueName: \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\") pod \"payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.239976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.239891 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerStarted","Data":"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5"} Oct 30 21:53:43.241766 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.241739 4525 generic.go:356] "Generic (PLEG): container finished" podID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerID="cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256" exitCode=0 Oct 30 21:53:43.241902 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.241804 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerDied","Data":"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256"} Oct 30 21:53:43.242215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.242200 4525 scope.go:122] "RemoveContainer" containerID="cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256" Oct 30 21:53:43.243964 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.243934 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:43.244051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.244006 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:43.244051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.244036 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:43.281381 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.281268 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:43.344490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.344341 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:53:43.345033 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.344888 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:43.345033 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.344967 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:43.722297 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:53:43.722257 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54b0950_0cef_44bb_b04b_677f0acbb6db.slice/crio-af89d0a9f9f02d8309c42add3c2bfa9390e82d1f58ce74cbfef2e73098ba02a8 WatchSource:0}: Error finding container af89d0a9f9f02d8309c42add3c2bfa9390e82d1f58ce74cbfef2e73098ba02a8: Status 404 returned error can't find the container with id af89d0a9f9f02d8309c42add3c2bfa9390e82d1f58ce74cbfef2e73098ba02a8 Oct 30 21:53:43.722884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:43.722801 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:53:44.306648 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.304006 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" event={"ID":"e54b0950-0cef-44bb-b04b-677f0acbb6db","Type":"ContainerStarted","Data":"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f"} Oct 30 21:53:44.306648 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.304078 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" event={"ID":"e54b0950-0cef-44bb-b04b-677f0acbb6db","Type":"ContainerStarted","Data":"af89d0a9f9f02d8309c42add3c2bfa9390e82d1f58ce74cbfef2e73098ba02a8"} Oct 30 21:53:44.306648 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.305299 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:53:44.310634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.309511 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerStarted","Data":"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315"} Oct 30 21:53:44.347067 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.347027 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:44.347360 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.347327 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:44.425143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:44.425089 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" podStartSLOduration=2.425074934 podStartE2EDuration="2.425074934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:44.422085312 +0000 UTC m=+1849.096131839" watchObservedRunningTime="2026-10-30 21:53:44.425074934 +0000 UTC m=+1849.099121464" Oct 30 21:53:45.272809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:45.272776 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:53:45.273687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:45.273661 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:53:45.273886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:45.273862 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:53:45.350633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:45.350405 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:45.350633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:45.350459 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:46.272167 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:46.272133 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:53:46.272470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:46.272439 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:53:46.357671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:46.350724 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:46.357671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:46.350787 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:46.721074 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:46.721026 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:53:47.273810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:47.273773 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:53:47.274085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:47.274057 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:53:47.351749 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:47.351715 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:47.352009 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:47.351983 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:48.271491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:48.271014 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:53:48.271491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:48.271089 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:53:48.379238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:48.379191 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:53:48.379569 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:48.379511 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:53:49.268040 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:49.268006 4525 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:53:49.268229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:49.268089 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 21:53:49.359307 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:49.359275 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:53:49.359825 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:49.359742 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:53:49.360737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:49.360714 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:53:50.272385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:50.272315 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:53:50.273595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:50.273575 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:53:56.325992 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:53:56.325963 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:54:05.398814 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:54:05.398775 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:54:09.849277 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:09.849243 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:54:09.849739 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:09.849453 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" containerID="cri-o://e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361" gracePeriod=30 Oct 30 21:54:10.886679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:10.875997 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:54:10.903286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:10.900958 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:10.927680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:10.927470 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:54:10.972651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:10.969922 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e51c52\"" Oct 30 21:54:10.972651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:10.970689 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e51c52-dockercfg-67f7f\"" Oct 30 21:54:11.044630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.008355 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.062631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.048502 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwk9\" (UniqueName: \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.096630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.079216 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:54:11.109667 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.109631 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.196258 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.196159 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:54:11.256659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.228628 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwk9\" (UniqueName: \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.257057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.257029 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgs4\" (UniqueName: \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.257363 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.257345 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.258297 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.258280 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.258711 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.258213 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.290812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.290779 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwk9\" (UniqueName: \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\") pod \"payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.359798 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.359760 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.360005 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.359985 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgs4\" (UniqueName: \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.360516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.360494 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.422257 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.422220 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgs4\" (UniqueName: \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\") pod \"payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.441182 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.441139 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:11.441335 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.441201 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:11.527693 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.527563 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:11.558769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.558730 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:11.914471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.914441 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:54:11.915196 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:54:11.915171 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec442a6_643a_4837_b8e5_3bac41636426.slice/crio-4c52e483270b19777245a3bd12be3e0ae99cd644bbcc9b1636f54d8fc0d9fcca WatchSource:0}: Error finding container 4c52e483270b19777245a3bd12be3e0ae99cd644bbcc9b1636f54d8fc0d9fcca: Status 404 returned error can't find the container with id 4c52e483270b19777245a3bd12be3e0ae99cd644bbcc9b1636f54d8fc0d9fcca Oct 30 21:54:11.968011 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:11.967977 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:54:11.968138 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:54:11.968021 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f4e0a_75d2_4f1e_85b9_089d78b6a2fa.slice/crio-8278e7f839c46476d5972a9f3a4e22609cce0f718fa759a2e4415561c756e04f WatchSource:0}: Error finding container 8278e7f839c46476d5972a9f3a4e22609cce0f718fa759a2e4415561c756e04f: Status 404 returned error can't find the container with id 8278e7f839c46476d5972a9f3a4e22609cce0f718fa759a2e4415561c756e04f Oct 30 21:54:12.754653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.754583 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" event={"ID":"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa","Type":"ContainerStarted","Data":"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760"} Oct 30 21:54:12.754875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.754683 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:12.754875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.754707 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" event={"ID":"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa","Type":"ContainerStarted","Data":"8278e7f839c46476d5972a9f3a4e22609cce0f718fa759a2e4415561c756e04f"} Oct 30 21:54:12.756460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.756424 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" event={"ID":"fec442a6-643a-4837-b8e5-3bac41636426","Type":"ContainerStarted","Data":"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d"} Oct 30 21:54:12.756575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.756469 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" event={"ID":"fec442a6-643a-4837-b8e5-3bac41636426","Type":"ContainerStarted","Data":"4c52e483270b19777245a3bd12be3e0ae99cd644bbcc9b1636f54d8fc0d9fcca"} Oct 30 21:54:12.756575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.756514 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:12.892850 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:12.892800 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" podStartSLOduration=2.89278425 podStartE2EDuration="2.89278425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.888484217 +0000 UTC m=+1877.562530746" watchObservedRunningTime="2026-10-30 21:54:12.89278425 +0000 UTC m=+1877.566830787" Oct 30 21:54:13.013684 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:13.011963 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" podStartSLOduration=3.011868468 podStartE2EDuration="3.011868468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:13.010792291 +0000 UTC m=+1877.684838829" watchObservedRunningTime="2026-10-30 21:54:13.011868468 +0000 UTC m=+1877.685915005" Oct 30 21:54:15.124691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.124668 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:54:15.223241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223211 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223267 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223361 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223388 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223414 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223472 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223574 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223635 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223707 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\") pod \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\" (UID: \"3aaa84ab-14e9-4977-96e5-eda7b4a3b955\") " Oct 30 21:54:15.223835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223729 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:15.223835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.223749 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:15.224081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224047 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:15.224081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224049 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:15.224216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224125 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:15.224479 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224454 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.224769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224481 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.224769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224491 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.224769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224502 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.224769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.224517 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.225814 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.225788 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:15.226202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.226177 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:15.226333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.226315 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:15.226386 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.226368 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj" pod "3aaa84ab-14e9-4977-96e5-eda7b4a3b955" (UID: "3aaa84ab-14e9-4977-96e5-eda7b4a3b955"). InnerVolumeSpecName "kube-api-access-5rpdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:15.326201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.326157 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.326201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.326191 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpdj\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-kube-api-access-5rpdj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.326201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.326201 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.326201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.326209 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aaa84ab-14e9-4977-96e5-eda7b4a3b955-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.778524 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.778431 4525 generic.go:356] "Generic (PLEG): container finished" podID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerID="e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361" exitCode=0 Oct 30 21:54:15.778524 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.778484 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" Oct 30 21:54:15.778797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.778516 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" event={"ID":"3aaa84ab-14e9-4977-96e5-eda7b4a3b955","Type":"ContainerDied","Data":"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361"} Oct 30 21:54:15.778797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.778583 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw" event={"ID":"3aaa84ab-14e9-4977-96e5-eda7b4a3b955","Type":"ContainerDied","Data":"8b09d46d8c7aece033ae1da6fbd39e96a495c4520c03d873e4eff0c1b05a9fc0"} Oct 30 21:54:15.778797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.778647 4525 scope.go:122] "RemoveContainer" containerID="e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361" Oct 30 21:54:15.789458 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.789440 4525 scope.go:122] "RemoveContainer" containerID="e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361" Oct 30 21:54:15.789789 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:54:15.789768 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361\": container with ID starting with e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361 not found: ID does not exist" containerID="e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361" Oct 30 21:54:15.789847 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.789798 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361"} err="failed to get container status \"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361\": rpc error: code = NotFound desc = could not find container \"e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361\": container with ID starting with e7f3b5bb859e6361f068b1c47f01e431a6c7896837ee1771aef82ff5e45f6361 not found: ID does not exist" Oct 30 21:54:15.948387 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.948355 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:54:15.957891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:15.957865 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-h7dmw"] Oct 30 21:54:17.476285 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:17.476256 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" path="/var/lib/kubelet/pods/3aaa84ab-14e9-4977-96e5-eda7b4a3b955/volumes" Oct 30 21:54:23.140116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.131827 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:54:23.140116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.133063 4525 cpu_manager.go:414] "containerMap: removing container" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" Oct 30 21:54:23.140116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.133084 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" Oct 30 21:54:23.140116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.133375 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaa84ab-14e9-4977-96e5-eda7b4a3b955" containerName="istio-proxy" Oct 30 21:54:23.143218 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.143191 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.195530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.195494 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:54:23.240145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.240112 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-79e10a-serving-cert\"" Oct 30 21:54:23.286698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.286665 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.286985 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.286960 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rfb\" (UniqueName: \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.287131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.287113 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.287393 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.287365 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.391941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.391843 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.392339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.392089 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rfb\" (UniqueName: \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.392339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.392198 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.392339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.392278 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.435628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.435160 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.449642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.444996 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.473634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.467227 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.510010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.509944 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rfb\" (UniqueName: \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\") pod \"maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.762716 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.762618 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:23.767436 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.767411 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:54:23.767670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:23.767500 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:54:24.149394 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:24.149364 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:54:24.149801 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:54:24.149775 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992e26f2_761b_4cf6_8a52_936db77eb417.slice/crio-02b5a727c5fb054b292ef121805768db43091e615810f6032e1f98c64dbf9c43 WatchSource:0}: Error finding container 02b5a727c5fb054b292ef121805768db43091e615810f6032e1f98c64dbf9c43: Status 404 returned error can't find the container with id 02b5a727c5fb054b292ef121805768db43091e615810f6032e1f98c64dbf9c43 Oct 30 21:54:24.957680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:24.957646 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerStarted","Data":"02b5a727c5fb054b292ef121805768db43091e615810f6032e1f98c64dbf9c43"} Oct 30 21:54:25.966923 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:25.966891 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerStarted","Data":"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624"} Oct 30 21:54:25.967333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:25.967026 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:26.090574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:26.090519 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" podStartSLOduration=2.124056144 podStartE2EDuration="3.090505438s" totalImagesPullingTime="966.449294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:26.087001342 +0000 UTC m=+1890.761047878" watchObservedRunningTime="2026-10-30 21:54:26.090505438 +0000 UTC m=+1890.764551975" Oct 30 21:54:28.860630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:28.860561 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:54:28.882103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:28.882066 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:28.903448 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:28.903418 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:54:28.979637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:28.970142 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5bfa09\"" Oct 30 21:54:28.979637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:28.970339 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5bfa09-dockercfg-tpx5b\"" Oct 30 21:54:29.033124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.016719 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjl4k\" (UniqueName: \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.033124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.016910 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.129636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.118562 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjl4k\" (UniqueName: \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.129636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.118842 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.129636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.120119 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.144837 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.143666 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:54:29.287041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.284912 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:54:29.287041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.285051 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.294587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.294433 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjl4k\" (UniqueName: \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\") pod \"payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.432259 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.432162 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.432461 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.432348 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wmll\" (UniqueName: \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.504580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.504535 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:29.534269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.534235 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.534432 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.534399 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wmll\" (UniqueName: \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.535263 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.535238 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.601691 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.598471 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wmll\" (UniqueName: \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\") pod \"payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.624101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.624071 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:29.885877 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:54:29.885842 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52b3140_9db4_4f51_84d4_ce72106b5bc7.slice/crio-5ffbbf62c0e18e5636daa70a8c64d32d97c25b2ae848575dcb517af1fe9b2fc1 WatchSource:0}: Error finding container 5ffbbf62c0e18e5636daa70a8c64d32d97c25b2ae848575dcb517af1fe9b2fc1: Status 404 returned error can't find the container with id 5ffbbf62c0e18e5636daa70a8c64d32d97c25b2ae848575dcb517af1fe9b2fc1 Oct 30 21:54:29.886232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:29.886197 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:54:30.021328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.019347 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:54:30.074789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.074758 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" event={"ID":"a52b3140-9db4-4f51-84d4-ce72106b5bc7","Type":"ContainerStarted","Data":"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df"} Oct 30 21:54:30.074942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.074812 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" event={"ID":"a52b3140-9db4-4f51-84d4-ce72106b5bc7","Type":"ContainerStarted","Data":"5ffbbf62c0e18e5636daa70a8c64d32d97c25b2ae848575dcb517af1fe9b2fc1"} Oct 30 21:54:30.074942 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.074857 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:30.076442 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.076417 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" event={"ID":"4797e46e-7da8-46f8-8b8f-77109e06e63f","Type":"ContainerStarted","Data":"e76b707a0047c2e7b4a766d818de021136f4ba1c702c9a16ed0f5b4b6e8d24f8"} Oct 30 21:54:30.200634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:30.200509 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" podStartSLOduration=2.200494901 podStartE2EDuration="2.200494901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:30.198304804 +0000 UTC m=+1894.872351341" watchObservedRunningTime="2026-10-30 21:54:30.200494901 +0000 UTC m=+1894.874541436" Oct 30 21:54:31.046432 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:31.046399 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:54:31.087710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:31.087657 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" event={"ID":"4797e46e-7da8-46f8-8b8f-77109e06e63f","Type":"ContainerStarted","Data":"5963c72a6f4075dcb384bc586bc5503d32e1c8184f7507edb8533df443e2ab6e"} Oct 30 21:54:31.087921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:31.087740 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:31.207634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:31.207356 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" podStartSLOduration=3.207337423 podStartE2EDuration="3.207337423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:31.204075568 +0000 UTC m=+1895.878122106" watchObservedRunningTime="2026-10-30 21:54:31.207337423 +0000 UTC m=+1895.881383959" Oct 30 21:54:31.975373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:31.975331 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.122:8443/health\": dial tcp 10.133.0.122:8443: connect: connection refused" Oct 30 21:54:35.399137 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:54:35.399096 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:54:36.134029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:36.133981 4525 generic.go:356] "Generic (PLEG): container finished" podID="992e26f2-761b-4cf6-8a52-936db77eb417" containerID="c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624" exitCode=1 Oct 30 21:54:36.134225 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:36.134064 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerDied","Data":"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624"} Oct 30 21:54:36.134515 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:36.134501 4525 scope.go:122] "RemoveContainer" containerID="c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624" Oct 30 21:54:37.142865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:37.142833 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerStarted","Data":"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e"} Oct 30 21:54:37.143237 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:37.143045 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:41.091550 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:41.091515 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:54:42.021236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.021202 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:54:42.030043 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.030013 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.051889 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.051857 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:54:42.116017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.115967 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-78c632-dockercfg-7bdw5\"" Oct 30 21:54:42.117379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.117352 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-78c632\"" Oct 30 21:54:42.120108 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.120062 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.121769 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.121739 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg76\" (UniqueName: \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.139496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.139460 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:54:42.266744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.264918 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.266744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.264994 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg76\" (UniqueName: \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.266744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.266460 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.338350 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.338236 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg76\" (UniqueName: \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\") pod \"payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.372470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.372209 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:42.774709 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:54:42.774671 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ca5322_49c1_4f2a_84f0_c2408ccfaa21.slice/crio-9cbccd1949aacca4d4c3afa8141143f3fa83a4b18440d7a8612c6e5d01801a85 WatchSource:0}: Error finding container 9cbccd1949aacca4d4c3afa8141143f3fa83a4b18440d7a8612c6e5d01801a85: Status 404 returned error can't find the container with id 9cbccd1949aacca4d4c3afa8141143f3fa83a4b18440d7a8612c6e5d01801a85 Oct 30 21:54:42.774856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:42.774834 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:54:43.156328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:43.156275 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:54:43.307976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:43.307939 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" event={"ID":"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21","Type":"ContainerStarted","Data":"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493"} Oct 30 21:54:43.308145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:43.308003 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" event={"ID":"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21","Type":"ContainerStarted","Data":"9cbccd1949aacca4d4c3afa8141143f3fa83a4b18440d7a8612c6e5d01801a85"} Oct 30 21:54:43.308145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:43.308052 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:54:43.446552 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:43.446441 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" podStartSLOduration=2.446426816 podStartE2EDuration="2.446426816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:43.442801308 +0000 UTC m=+1908.116847845" watchObservedRunningTime="2026-10-30 21:54:43.446426816 +0000 UTC m=+1908.120473356" Oct 30 21:54:54.318584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:54:54.318552 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:55:02.335528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:02.335496 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:55:02.470402 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:02.470356 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:55:05.405850 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:05.405810 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:55:07.336706 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.336595 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" podUID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" containerName="payload-processing" containerID="cri-o://5fe2df643dc4489ea1b1b4c5539e08c4af35a8770b5c03d42818861a7c1c7cc5" gracePeriod=25 Oct 30 21:55:07.484145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.484107 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" podUID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" containerName="payload-pre-processing" containerID="cri-o://2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013" gracePeriod=25 Oct 30 21:55:07.521052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.521017 4525 generic.go:356] "Generic (PLEG): container finished" podID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" containerID="5fe2df643dc4489ea1b1b4c5539e08c4af35a8770b5c03d42818861a7c1c7cc5" exitCode=0 Oct 30 21:55:07.521201 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.521048 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" event={"ID":"9bc7c0f7-b233-422b-ad24-2fd18c28000c","Type":"ContainerDied","Data":"5fe2df643dc4489ea1b1b4c5539e08c4af35a8770b5c03d42818861a7c1c7cc5"} Oct 30 21:55:07.702494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.701933 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:55:07.805934 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.805909 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:55:07.825775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.825746 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\") pod \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " Oct 30 21:55:07.825920 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.825815 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\" (UniqueName: \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\") pod \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\" (UID: \"9bc7c0f7-b233-422b-ad24-2fd18c28000c\") " Oct 30 21:55:07.826113 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.826090 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume" pod "9bc7c0f7-b233-422b-ad24-2fd18c28000c" (UID: "9bc7c0f7-b233-422b-ad24-2fd18c28000c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:07.826379 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.826364 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc7c0f7-b233-422b-ad24-2fd18c28000c-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:07.828091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.828066 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv" pod "9bc7c0f7-b233-422b-ad24-2fd18c28000c" (UID: "9bc7c0f7-b233-422b-ad24-2fd18c28000c"). InnerVolumeSpecName "kube-api-access-f8tjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:07.927705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.927517 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\") pod \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " Oct 30 21:55:07.927891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.927800 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\" (UniqueName: \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\") pod \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\" (UID: \"d6f97e24-1cbd-40e7-876d-b4e6d77e5606\") " Oct 30 21:55:07.928908 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.928880 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tjv\" (UniqueName: \"kubernetes.io/projected/9bc7c0f7-b233-422b-ad24-2fd18c28000c-kube-api-access-f8tjv\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:07.931787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.931754 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf" pod "d6f97e24-1cbd-40e7-876d-b4e6d77e5606" (UID: "d6f97e24-1cbd-40e7-876d-b4e6d77e5606"). InnerVolumeSpecName "kube-api-access-8mrlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:07.932121 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:07.932098 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume" pod "d6f97e24-1cbd-40e7-876d-b4e6d77e5606" (UID: "d6f97e24-1cbd-40e7-876d-b4e6d77e5606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:08.030176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.030138 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mrlf\" (UniqueName: \"kubernetes.io/projected/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-kube-api-access-8mrlf\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:08.030176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.030169 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f97e24-1cbd-40e7-876d-b4e6d77e5606-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:08.542859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.542775 4525 generic.go:356] "Generic (PLEG): container finished" podID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" containerID="2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013" exitCode=0 Oct 30 21:55:08.543862 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.543834 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" Oct 30 21:55:08.545097 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.544039 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" event={"ID":"d6f97e24-1cbd-40e7-876d-b4e6d77e5606","Type":"ContainerDied","Data":"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013"} Oct 30 21:55:08.545213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.545150 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk" event={"ID":"d6f97e24-1cbd-40e7-876d-b4e6d77e5606","Type":"ContainerDied","Data":"8fde64a8b86195250c73db4d40a01dcae8c489caf40df4700d754bee6fb0eacb"} Oct 30 21:55:08.545213 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.545191 4525 scope.go:122] "RemoveContainer" containerID="2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013" Oct 30 21:55:08.546135 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.546109 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" event={"ID":"9bc7c0f7-b233-422b-ad24-2fd18c28000c","Type":"ContainerDied","Data":"69775aa47aadf09bb1cf4504513322c444034a027ec30d3c7df356cc59ebe07b"} Oct 30 21:55:08.546731 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.546680 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz" Oct 30 21:55:08.558292 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.558275 4525 scope.go:122] "RemoveContainer" containerID="2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013" Oct 30 21:55:08.558565 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:08.558548 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013\": container with ID starting with 2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013 not found: ID does not exist" containerID="2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013" Oct 30 21:55:08.558639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.558576 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013"} err="failed to get container status \"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013\": rpc error: code = NotFound desc = could not find container \"2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013\": container with ID starting with 2aa215190acf812c26583f3c01d4b55e2e372bbdca3787c6b7b656885cddd013 not found: ID does not exist" Oct 30 21:55:08.558639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.558594 4525 scope.go:122] "RemoveContainer" containerID="5fe2df643dc4489ea1b1b4c5539e08c4af35a8770b5c03d42818861a7c1c7cc5" Oct 30 21:55:08.712506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.712467 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:55:08.768810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.768775 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8709fb3-7985994f8b-zd5cz"] Oct 30 21:55:08.828877 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.828846 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:55:08.833516 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:08.833491 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8709fb3-967b645f9-jbdbk"] Oct 30 21:55:09.476377 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:09.476343 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" path="/var/lib/kubelet/pods/9bc7c0f7-b233-422b-ad24-2fd18c28000c/volumes" Oct 30 21:55:09.476871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:09.476856 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" path="/var/lib/kubelet/pods/d6f97e24-1cbd-40e7-876d-b4e6d77e5606/volumes" Oct 30 21:55:21.988277 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:21.988219 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:55:21.988891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:21.988470 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" containerID="cri-o://38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a" gracePeriod=30 Oct 30 21:55:22.026995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.026963 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:55:22.027208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.027168 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" containerID="cri-o://4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9" gracePeriod=30 Oct 30 21:55:22.036038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.036013 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:55:22.036221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.036190 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" containerID="cri-o://500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f" gracePeriod=30 Oct 30 21:55:22.039523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.039500 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:55:22.039698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.039679 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" containerID="cri-o://b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53" gracePeriod=30 Oct 30 21:55:22.065978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:22.065948 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:55:27.357440 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.357119 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:55:27.389267 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389226 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389436 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389278 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389436 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389325 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389449 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389531 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389596 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389722 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.389860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389837 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.390029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.389865 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\") pod \"b804e39b-fd87-411b-801a-c273b1cdeb45\" (UID: \"b804e39b-fd87-411b-801a-c273b1cdeb45\") " Oct 30 21:55:27.390835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.390806 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.391990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.391679 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.391990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.391846 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.391990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.391878 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:27.392207 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.392015 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.394029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.393994 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.394124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.394075 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.395414 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.395378 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "kube-api-access-4tncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.415804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.415753 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo" pod "b804e39b-fd87-411b-801a-c273b1cdeb45" (UID: "b804e39b-fd87-411b-801a-c273b1cdeb45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:27.444196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.444167 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:55:27.451123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.451099 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:55:27.456062 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.456041 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:55:27.491641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491580 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.491641 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491634 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491669 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491698 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491720 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491738 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491758 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.491812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491801 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491828 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491837 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491951 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.491992 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492023 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492065 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492093 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492094 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.492141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492134 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492171 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492202 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492227 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492241 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492271 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492305 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492334 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492376 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492411 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492421 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492464 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492481 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492508 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\") pod \"b48457e2-c981-400f-8115-29e53b06c69a\" (UID: \"b48457e2-c981-400f-8115-29e53b06c69a\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492543 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492570 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492622 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\") pod \"0c035e29-2266-4243-90e7-620553f3087b\" (UID: \"0c035e29-2266-4243-90e7-620553f3087b\") " Oct 30 21:55:27.492644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492637 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.493464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492672 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.493464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492774 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.493464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.492865 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\") pod \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\" (UID: \"49c00a89-b44d-4daf-a51e-7ae5e3d09a70\") " Oct 30 21:55:27.493667 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493649 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493675 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493690 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493705 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493719 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tncj\" (UniqueName: \"kubernetes.io/projected/b804e39b-fd87-411b-801a-c273b1cdeb45-kube-api-access-4tncj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493733 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.493747 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493746 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493761 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493775 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48457e2-c981-400f-8115-29e53b06c69a-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493790 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493805 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493818 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493832 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493845 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b804e39b-fd87-411b-801a-c273b1cdeb45-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493859 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b804e39b-fd87-411b-801a-c273b1cdeb45-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493872 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493886 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493900 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494047 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.493913 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b804e39b-fd87-411b-801a-c273b1cdeb45-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.494737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.494585 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:27.494737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.494591 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.494961 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.494935 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:27.495189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495164 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.495256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495165 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:27.495256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495203 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.495256 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495219 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.495359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495253 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.495428 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495406 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.495805 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.495777 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:27.496202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.496177 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.497916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.497884 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.498003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.497935 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "kube-api-access-rfl5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.498003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.497969 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:27.498137 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.498119 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44" pod "b48457e2-c981-400f-8115-29e53b06c69a" (UID: "b48457e2-c981-400f-8115-29e53b06c69a"). InnerVolumeSpecName "kube-api-access-cfl44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.498239 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.498221 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy" pod "49c00a89-b44d-4daf-a51e-7ae5e3d09a70" (UID: "49c00a89-b44d-4daf-a51e-7ae5e3d09a70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.498355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.498335 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw" pod "0c035e29-2266-4243-90e7-620553f3087b" (UID: "0c035e29-2266-4243-90e7-620553f3087b"). InnerVolumeSpecName "kube-api-access-qj9jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595235 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595263 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595273 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48457e2-c981-400f-8115-29e53b06c69a-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595288 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfl44\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-kube-api-access-cfl44\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595298 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595308 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595316 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595316 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9jw\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-kube-api-access-qj9jw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595324 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595332 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfl5n\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-kube-api-access-rfl5n\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595340 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595348 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c035e29-2266-4243-90e7-620553f3087b-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595361 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48457e2-c981-400f-8115-29e53b06c69a-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595374 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48457e2-c981-400f-8115-29e53b06c69a-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595381 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c00a89-b44d-4daf-a51e-7ae5e3d09a70-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595390 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c035e29-2266-4243-90e7-620553f3087b-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595397 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c035e29-2266-4243-90e7-620553f3087b-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.595698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.595406 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c035e29-2266-4243-90e7-620553f3087b-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:27.751667 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.751636 4525 generic.go:356] "Generic (PLEG): container finished" podID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerID="b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53" exitCode=0 Oct 30 21:55:27.751863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.751684 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" Oct 30 21:55:27.751863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.751707 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" event={"ID":"49c00a89-b44d-4daf-a51e-7ae5e3d09a70","Type":"ContainerDied","Data":"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53"} Oct 30 21:55:27.751863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.751761 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4" event={"ID":"49c00a89-b44d-4daf-a51e-7ae5e3d09a70","Type":"ContainerDied","Data":"3f8c3263cd6b99f9e411184dabb6e9f0c995519f6cab5c3b1f9edf6337475f3e"} Oct 30 21:55:27.751863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.751783 4525 scope.go:122] "RemoveContainer" containerID="b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53" Oct 30 21:55:27.753216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.753191 4525 generic.go:356] "Generic (PLEG): container finished" podID="b48457e2-c981-400f-8115-29e53b06c69a" containerID="4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9" exitCode=0 Oct 30 21:55:27.753343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.753241 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" Oct 30 21:55:27.753343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.753270 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" event={"ID":"b48457e2-c981-400f-8115-29e53b06c69a","Type":"ContainerDied","Data":"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9"} Oct 30 21:55:27.753343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.753329 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn" event={"ID":"b48457e2-c981-400f-8115-29e53b06c69a","Type":"ContainerDied","Data":"863297a4be03d6bae38cdf0e04fbfeb6efba8e62f2f277165ac952ed30bd4790"} Oct 30 21:55:27.754891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.754866 4525 generic.go:356] "Generic (PLEG): container finished" podID="0c035e29-2266-4243-90e7-620553f3087b" containerID="38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a" exitCode=0 Oct 30 21:55:27.755029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.754909 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" Oct 30 21:55:27.755029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.754912 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" event={"ID":"0c035e29-2266-4243-90e7-620553f3087b","Type":"ContainerDied","Data":"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a"} Oct 30 21:55:27.755029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.754964 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr" event={"ID":"0c035e29-2266-4243-90e7-620553f3087b","Type":"ContainerDied","Data":"19ba9ffe641ee9b7599dcea11b5b4e2be1e1fb40d41bdeb43fa15a36641fbe04"} Oct 30 21:55:27.756811 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.756783 4525 generic.go:356] "Generic (PLEG): container finished" podID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerID="500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f" exitCode=0 Oct 30 21:55:27.756955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.756873 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" event={"ID":"b804e39b-fd87-411b-801a-c273b1cdeb45","Type":"ContainerDied","Data":"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f"} Oct 30 21:55:27.756955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.756917 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" Oct 30 21:55:27.757198 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.756912 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq" event={"ID":"b804e39b-fd87-411b-801a-c273b1cdeb45","Type":"ContainerDied","Data":"399c8da17c9d38260a05d40c333edd459066f4e655e2a8a33f8128a6555fa265"} Oct 30 21:55:27.765840 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.765822 4525 scope.go:122] "RemoveContainer" containerID="b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53" Oct 30 21:55:27.766259 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:27.766231 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53\": container with ID starting with b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53 not found: ID does not exist" containerID="b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53" Oct 30 21:55:27.766359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.766269 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53"} err="failed to get container status \"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53\": rpc error: code = NotFound desc = could not find container \"b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53\": container with ID starting with b6fa0911b4b104e02b642c3acb4dd0091377acc5cc5769adf291a62866637b53 not found: ID does not exist" Oct 30 21:55:27.766359 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.766292 4525 scope.go:122] "RemoveContainer" containerID="4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9" Oct 30 21:55:27.776138 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.776120 4525 scope.go:122] "RemoveContainer" containerID="4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9" Oct 30 21:55:27.776412 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:27.776389 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9\": container with ID starting with 4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9 not found: ID does not exist" containerID="4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9" Oct 30 21:55:27.776485 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.776423 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9"} err="failed to get container status \"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9\": rpc error: code = NotFound desc = could not find container \"4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9\": container with ID starting with 4d5cc462303090326d8a929a30841aa9c35e3f80ee137a2b0f2c328617fb77b9 not found: ID does not exist" Oct 30 21:55:27.776485 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.776446 4525 scope.go:122] "RemoveContainer" containerID="38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a" Oct 30 21:55:27.799178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.799154 4525 scope.go:122] "RemoveContainer" containerID="38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a" Oct 30 21:55:27.799659 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:27.799596 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a\": container with ID starting with 38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a not found: ID does not exist" containerID="38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a" Oct 30 21:55:27.799779 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.799674 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a"} err="failed to get container status \"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a\": rpc error: code = NotFound desc = could not find container \"38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a\": container with ID starting with 38334e4ad4389bf6c9f459fca456035d6baef7c3df57e32868012f5daa98d74a not found: ID does not exist" Oct 30 21:55:27.799779 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.799704 4525 scope.go:122] "RemoveContainer" containerID="500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f" Oct 30 21:55:27.814176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.814148 4525 scope.go:122] "RemoveContainer" containerID="500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f" Oct 30 21:55:27.814526 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:27.814503 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f\": container with ID starting with 500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f not found: ID does not exist" containerID="500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f" Oct 30 21:55:27.814682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:27.814539 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f"} err="failed to get container status \"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f\": rpc error: code = NotFound desc = could not find container \"500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f\": container with ID starting with 500d76ad23e9a26cf5d8a5ad32dc7fa27401b717987df45c3f33da651f4d9f7f not found: ID does not exist" Oct 30 21:55:28.067881 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.067846 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:55:28.075754 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.075722 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-89fbn"] Oct 30 21:55:28.196960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.196922 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:55:28.198655 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.198627 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fxrfr"] Oct 30 21:55:28.309439 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.309356 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:55:28.317971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.317943 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-d88rq"] Oct 30 21:55:28.422226 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.422195 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:55:28.432143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:28.432117 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-48np4"] Oct 30 21:55:29.474501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:29.474467 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c035e29-2266-4243-90e7-620553f3087b" path="/var/lib/kubelet/pods/0c035e29-2266-4243-90e7-620553f3087b/volumes" Oct 30 21:55:29.475072 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:29.475054 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" path="/var/lib/kubelet/pods/49c00a89-b44d-4daf-a51e-7ae5e3d09a70/volumes" Oct 30 21:55:29.475510 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:29.475491 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48457e2-c981-400f-8115-29e53b06c69a" path="/var/lib/kubelet/pods/b48457e2-c981-400f-8115-29e53b06c69a/volumes" Oct 30 21:55:29.476295 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:29.476276 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" path="/var/lib/kubelet/pods/b804e39b-fd87-411b-801a-c273b1cdeb45/volumes" Oct 30 21:55:35.397592 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:35.397555 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:55:35.416593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.416561 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:55:35.417403 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417385 4525 cpu_manager.go:414] "containerMap: removing container" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417405 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417434 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417441 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417456 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417465 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417474 4525 cpu_manager.go:414] "containerMap: removing container" podUID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" containerName="payload-processing" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417479 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" containerName="payload-processing" Oct 30 21:55:35.417496 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417498 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" containerName="payload-pre-processing" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417503 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" containerName="payload-pre-processing" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417513 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417518 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417649 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f97e24-1cbd-40e7-876d-b4e6d77e5606" containerName="payload-pre-processing" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417663 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b804e39b-fd87-411b-801a-c273b1cdeb45" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417673 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c00a89-b44d-4daf-a51e-7ae5e3d09a70" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417680 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c035e29-2266-4243-90e7-620553f3087b" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417689 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48457e2-c981-400f-8115-29e53b06c69a" containerName="istio-proxy" Oct 30 21:55:35.417888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.417695 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc7c0f7-b233-422b-ad24-2fd18c28000c" containerName="payload-processing" Oct 30 21:55:35.423133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.423116 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.483917 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.483889 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e51c52-serving-cert\"" Oct 30 21:55:35.508985 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.508945 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.509341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.509161 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.509341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.509307 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.509472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.509438 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.538142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.538103 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:55:35.611459 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.611426 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.611683 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.611540 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.611683 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:35.611573 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-serving-cert: secret "maas-api-e2e-api-a-e51c52-serving-cert" not found Oct 30 21:55:35.611683 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.611663 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.611683 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:35.611675 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls podName:86a87117-4564-47a5-a067-8d072b119395 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:36.111657809 +0000 UTC m=+1960.785704323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls") pod "maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" (UID: "86a87117-4564-47a5-a067-8d072b119395") : secret "maas-api-e2e-api-a-e51c52-serving-cert" not found Oct 30 21:55:35.611929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.611760 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.612354 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.612329 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.614351 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.614332 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:35.733334 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:35.733213 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:36.136024 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:36.135994 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:36.138731 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:36.138709 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") pod \"maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:36.337084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:36.337051 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:36.729073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:36.729044 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:55:36.731635 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:55:36.731588 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a87117_4564_47a5_a067_8d072b119395.slice/crio-3fcdc80937eafe18f7a1ec58c394bd18919d47164d7fd14a49843dd773e5eb6c WatchSource:0}: Error finding container 3fcdc80937eafe18f7a1ec58c394bd18919d47164d7fd14a49843dd773e5eb6c: Status 404 returned error can't find the container with id 3fcdc80937eafe18f7a1ec58c394bd18919d47164d7fd14a49843dd773e5eb6c Oct 30 21:55:36.824702 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:36.824667 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" event={"ID":"86a87117-4564-47a5-a067-8d072b119395","Type":"ContainerStarted","Data":"3fcdc80937eafe18f7a1ec58c394bd18919d47164d7fd14a49843dd773e5eb6c"} Oct 30 21:55:37.833914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:37.833828 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" event={"ID":"86a87117-4564-47a5-a067-8d072b119395","Type":"ContainerStarted","Data":"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8"} Oct 30 21:55:37.833914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:37.833893 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:38.008537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:38.008460 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" podStartSLOduration=2.23843428 podStartE2EDuration="3.008442247s" totalImagesPullingTime="770.007967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:38.007853391 +0000 UTC m=+1962.681900137" watchObservedRunningTime="2026-10-30 21:55:38.008442247 +0000 UTC m=+1962.682488783" Oct 30 21:55:42.120071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.120038 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7088-326bad"] Oct 30 21:55:42.124533 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.124516 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:42.190598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.190561 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7088-326bad"] Oct 30 21:55:42.222832 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.222797 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncw9l\"" Oct 30 21:55:42.224494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.224463 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzd8\" (UniqueName: \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\") pod \"test-curl-7088-326bad\" (UID: \"39695b4c-822d-4c49-874c-d27eed91fae2\") " pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:42.325479 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.325449 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzd8\" (UniqueName: \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\") pod \"test-curl-7088-326bad\" (UID: \"39695b4c-822d-4c49-874c-d27eed91fae2\") " pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:42.387678 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.387591 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzd8\" (UniqueName: \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\") pod \"test-curl-7088-326bad\" (UID: \"39695b4c-822d-4c49-874c-d27eed91fae2\") " pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:42.437058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.437025 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:42.816674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.816641 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7088-326bad"] Oct 30 21:55:42.817047 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:55:42.817023 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39695b4c_822d_4c49_874c_d27eed91fae2.slice/crio-8c2712c0d1c100414d76c50a156fd5cd0629722b749471e7d2e68ed6f87b5912 WatchSource:0}: Error finding container 8c2712c0d1c100414d76c50a156fd5cd0629722b749471e7d2e68ed6f87b5912: Status 404 returned error can't find the container with id 8c2712c0d1c100414d76c50a156fd5cd0629722b749471e7d2e68ed6f87b5912 Oct 30 21:55:42.879415 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:42.879380 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7088-326bad" event={"ID":"39695b4c-822d-4c49-874c-d27eed91fae2","Type":"ContainerStarted","Data":"8c2712c0d1c100414d76c50a156fd5cd0629722b749471e7d2e68ed6f87b5912"} Oct 30 21:55:43.847410 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:43.847380 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:55:46.912564 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:46.912526 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7088-326bad" event={"ID":"39695b4c-822d-4c49-874c-d27eed91fae2","Type":"ContainerStarted","Data":"4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367"} Oct 30 21:55:47.030530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:47.030472 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7088-326bad" podStartSLOduration=1.658828054 podStartE2EDuration="5.030456901s" totalImagesPullingTime="3.371628847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:47.027172648 +0000 UTC m=+1971.701219184" watchObservedRunningTime="2026-10-30 21:55:47.030456901 +0000 UTC m=+1971.704503438" Oct 30 21:55:48.616854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:48.616820 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7088-326bad"] Oct 30 21:55:48.623810 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:48.623779 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7088-326bad"] Oct 30 21:55:48.927323 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:48.927196 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7088-326bad" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" containerName="test-curl-7088-326bad" containerID="cri-o://4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367" gracePeriod=2 Oct 30 21:55:48.979451 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:48.979411 4525 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7088-326bad\" is forbidden: User \"system:node:ip-10-0-134-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-53.us-west-1.compute.internal' and this object" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:51.248058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.248035 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:51.364328 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.364290 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\" (UniqueName: \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\") pod \"39695b4c-822d-4c49-874c-d27eed91fae2\" (UID: \"39695b4c-822d-4c49-874c-d27eed91fae2\") " Oct 30 21:55:51.366860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.366831 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8" pod "39695b4c-822d-4c49-874c-d27eed91fae2" (UID: "39695b4c-822d-4c49-874c-d27eed91fae2"). InnerVolumeSpecName "kube-api-access-qwzd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:51.467112 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.467082 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzd8\" (UniqueName: \"kubernetes.io/projected/39695b4c-822d-4c49-874c-d27eed91fae2-kube-api-access-qwzd8\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:51.484152 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.484073 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" path="/var/lib/kubelet/pods/39695b4c-822d-4c49-874c-d27eed91fae2/volumes" Oct 30 21:55:51.950765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.950734 4525 generic.go:356] "Generic (PLEG): container finished" podID="39695b4c-822d-4c49-874c-d27eed91fae2" containerID="4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367" exitCode=137 Oct 30 21:55:51.950977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.950798 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-326bad" Oct 30 21:55:51.950977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.950826 4525 scope.go:122] "RemoveContainer" containerID="4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367" Oct 30 21:55:51.965031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.965006 4525 scope.go:122] "RemoveContainer" containerID="4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367" Oct 30 21:55:51.965426 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:51.965395 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367\": container with ID starting with 4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367 not found: ID does not exist" containerID="4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367" Oct 30 21:55:51.965492 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:51.965436 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367"} err="failed to get container status \"4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367\": rpc error: code = NotFound desc = could not find container \"4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367\": container with ID starting with 4b6a7aa516fe61685fb4106ec819cc9045765f7a26bcafe9aa50877b73f48367 not found: ID does not exist" Oct 30 21:55:56.055387 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.055351 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:55:56.056455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.056437 4525 cpu_manager.go:414] "containerMap: removing container" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" containerName="test-curl-7088-326bad" Oct 30 21:55:56.056514 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.056466 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" containerName="test-curl-7088-326bad" Oct 30 21:55:56.056699 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.056685 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="39695b4c-822d-4c49-874c-d27eed91fae2" containerName="test-curl-7088-326bad" Oct 30 21:55:56.062453 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.062433 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.115081 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.115049 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-5bfa09-serving-cert\"" Oct 30 21:55:56.131776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.131740 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.131951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.131789 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdv4\" (UniqueName: \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.132002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.131962 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.132049 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.132022 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.173669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.173504 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:55:56.234060 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.234009 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.234254 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.234200 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.234336 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.234264 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdv4\" (UniqueName: \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.234399 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:56.234331 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-serving-cert: secret "maas-api-e2e-shared-b-w1-5bfa09-serving-cert" not found Oct 30 21:55:56.234459 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:55:56.234420 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls podName:6de079fd-0bbd-4b58-864a-8a1d461245f5 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:56.734396955 +0000 UTC m=+1981.408443483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls") pod "maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" (UID: "6de079fd-0bbd-4b58-864a-8a1d461245f5") : secret "maas-api-e2e-shared-b-w1-5bfa09-serving-cert" not found Oct 30 21:55:56.234530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.234490 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.234841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.234816 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.237208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.237183 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.318682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.317571 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdv4\" (UniqueName: \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.749911 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.749874 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.752542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.752511 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:56.978205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:56.978170 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:57.368879 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:57.368850 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:55:57.370432 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:55:57.370405 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de079fd_0bbd_4b58_864a_8a1d461245f5.slice/crio-7ccb5e72754c53bcbd85bb64d83850b317a0492fa8058b870ea031f9130878fa WatchSource:0}: Error finding container 7ccb5e72754c53bcbd85bb64d83850b317a0492fa8058b870ea031f9130878fa: Status 404 returned error can't find the container with id 7ccb5e72754c53bcbd85bb64d83850b317a0492fa8058b870ea031f9130878fa Oct 30 21:55:57.997682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:57.997651 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerStarted","Data":"7ccb5e72754c53bcbd85bb64d83850b317a0492fa8058b870ea031f9130878fa"} Oct 30 21:55:59.006968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:59.006929 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerStarted","Data":"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c"} Oct 30 21:55:59.007380 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:59.007121 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:55:59.128087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:55:59.128026 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" podStartSLOduration=2.347833307 podStartE2EDuration="3.12801034s" totalImagesPullingTime="780.177033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:59.125395098 +0000 UTC m=+1983.799441639" watchObservedRunningTime="2026-10-30 21:55:59.12801034 +0000 UTC m=+1983.802056873" Oct 30 21:56:05.016919 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:05.016871 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": dial tcp 10.133.0.128:8443: connect: connection refused" Oct 30 21:56:05.397730 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:05.397691 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:56:09.107488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:09.107456 4525 generic.go:356] "Generic (PLEG): container finished" podID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerID="e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c" exitCode=1 Oct 30 21:56:09.107899 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:09.107524 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerDied","Data":"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c"} Oct 30 21:56:09.107979 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:09.107965 4525 scope.go:122] "RemoveContainer" containerID="e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c" Oct 30 21:56:10.116544 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:10.116499 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerStarted","Data":"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9"} Oct 30 21:56:10.116956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:10.116762 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:56:11.002790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:11.002756 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:56:11.003035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:11.002996 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" containerID="cri-o://0a8ed2404770fba0ccd1ea2db67398fda888997851746e8f29e5cb2e92544a0e" gracePeriod=30 Oct 30 21:56:13.313117 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.313082 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:56:13.317784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.317761 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.370278 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.370244 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-78c632-serving-cert\"" Oct 30 21:56:13.389886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.389849 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9dl\" (UniqueName: \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.390103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.389911 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.390103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.390002 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.390103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.390080 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.425248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.425198 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:56:13.493001 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.492962 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.493209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.493042 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.493209 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.493167 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.493362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.493345 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9dl\" (UniqueName: \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.493850 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:13.493827 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-serving-cert: secret "maas-api-e2e-shared-b-w4-78c632-serving-cert" not found Oct 30 21:56:13.494100 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:13.494085 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls podName:069c580a-1bb3-4bf6-a344-0c54b3145a30 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:13.99406177 +0000 UTC m=+1998.668108300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls") pod "maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" (UID: "069c580a-1bb3-4bf6-a344-0c54b3145a30") : secret "maas-api-e2e-shared-b-w4-78c632-serving-cert" not found Oct 30 21:56:13.494551 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.494524 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.499341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.499311 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:13.556583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:13.556554 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9dl\" (UniqueName: \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:14.015978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:14.015941 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:14.018716 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:14.018690 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:14.232721 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:14.232690 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:14.629762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:14.629732 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:56:14.630143 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:56:14.629894 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069c580a_1bb3_4bf6_a344_0c54b3145a30.slice/crio-a89fbaff315051f29d5c0395f36b1138cd82b1ef0828afaa1325b146555634d7 WatchSource:0}: Error finding container a89fbaff315051f29d5c0395f36b1138cd82b1ef0828afaa1325b146555634d7: Status 404 returned error can't find the container with id a89fbaff315051f29d5c0395f36b1138cd82b1ef0828afaa1325b146555634d7 Oct 30 21:56:15.157581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:15.157544 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" event={"ID":"069c580a-1bb3-4bf6-a344-0c54b3145a30","Type":"ContainerStarted","Data":"a89fbaff315051f29d5c0395f36b1138cd82b1ef0828afaa1325b146555634d7"} Oct 30 21:56:16.136176 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.136152 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:56:16.166333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.166303 4525 generic.go:356] "Generic (PLEG): container finished" podID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerID="0a8ed2404770fba0ccd1ea2db67398fda888997851746e8f29e5cb2e92544a0e" exitCode=0 Oct 30 21:56:16.166515 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.166380 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" event={"ID":"cf309456-53e5-4c32-a46d-4155dcc35dbb","Type":"ContainerDied","Data":"0a8ed2404770fba0ccd1ea2db67398fda888997851746e8f29e5cb2e92544a0e"} Oct 30 21:56:16.168296 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.168272 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" event={"ID":"069c580a-1bb3-4bf6-a344-0c54b3145a30","Type":"ContainerStarted","Data":"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a"} Oct 30 21:56:16.168443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.168334 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:16.321326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.321297 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:56:16.357463 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357436 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357518 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357671 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357778 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357844 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357940 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.357982 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358016 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358071 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358078 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.358325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358089 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\") pod \"cf309456-53e5-4c32-a46d-4155dcc35dbb\" (UID: \"cf309456-53e5-4c32-a46d-4155dcc35dbb\") " Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358361 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358439 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358719 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358912 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358925 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358935 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358944 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf309456-53e5-4c32-a46d-4155dcc35dbb-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.359130 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.358954 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.360000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.359973 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "kube-api-access-9k99k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:16.360248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.360221 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:16.360763 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.360739 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:16.360892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.360871 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy" pod "cf309456-53e5-4c32-a46d-4155dcc35dbb" (UID: "cf309456-53e5-4c32-a46d-4155dcc35dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:16.386151 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.386100 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" podStartSLOduration=2.572401386 podStartE2EDuration="3.386084815s" totalImagesPullingTime="813.683429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:16.385380837 +0000 UTC m=+2001.059427409" watchObservedRunningTime="2026-10-30 21:56:16.386084815 +0000 UTC m=+2001.060131350" Oct 30 21:56:16.460582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.460549 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.460582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.460577 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.460582 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.460587 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k99k\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-kube-api-access-9k99k\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:16.460842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:16.460596 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf309456-53e5-4c32-a46d-4155dcc35dbb-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:17.176786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.176756 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" Oct 30 21:56:17.177255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.176790 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g" event={"ID":"cf309456-53e5-4c32-a46d-4155dcc35dbb","Type":"ContainerDied","Data":"3a6fa2da4a339cce9b08829c1713961e69351cdbd53ce3d49b50b6460841a56d"} Oct 30 21:56:17.177255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.176866 4525 scope.go:122] "RemoveContainer" containerID="0a8ed2404770fba0ccd1ea2db67398fda888997851746e8f29e5cb2e92544a0e" Oct 30 21:56:17.348519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.348478 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:56:17.398668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.398634 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-pmx2g"] Oct 30 21:56:17.484865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.479962 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" path="/var/lib/kubelet/pods/cf309456-53e5-4c32-a46d-4155dcc35dbb/volumes" Oct 30 21:56:17.858430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.858347 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:56:17.858642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:17.858555 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" containerID="cri-o://7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441" gracePeriod=30 Oct 30 21:56:18.138661 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:18.138249 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:56:22.182637 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:22.182580 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:56:23.135495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.135414 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:56:23.139480 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.139446 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" podUID="dd2164db-f8c6-4d60-8089-55394bb918ea" containerName="payload-pre-processing" containerID="cri-o://ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2" gracePeriod=25 Oct 30 21:56:23.231195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.231161 4525 generic.go:356] "Generic (PLEG): container finished" podID="e7554552-a966-45ce-a189-7abd23f315db" containerID="7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441" exitCode=0 Oct 30 21:56:23.231595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.231209 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" Oct 30 21:56:23.231595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.231239 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerDied","Data":"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441"} Oct 30 21:56:23.231595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.231294 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm" event={"ID":"e7554552-a966-45ce-a189-7abd23f315db","Type":"ContainerDied","Data":"9af8268f2be5e06269d0024ad03604729a18f736e0cbaea26633bbc3649b978d"} Oct 30 21:56:23.231595 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.231318 4525 scope.go:122] "RemoveContainer" containerID="7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441" Oct 30 21:56:23.252540 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.252481 4525 scope.go:122] "RemoveContainer" containerID="8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece" Oct 30 21:56:23.263884 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.263861 4525 scope.go:122] "RemoveContainer" containerID="7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441" Oct 30 21:56:23.264220 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:23.264196 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441\": container with ID starting with 7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441 not found: ID does not exist" containerID="7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441" Oct 30 21:56:23.264298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.264228 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441"} err="failed to get container status \"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441\": rpc error: code = NotFound desc = could not find container \"7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441\": container with ID starting with 7bf6bfb8087ed74a4cf7d27c67715c35aea2ed0ee6814c7890b1eed65b4fd441 not found: ID does not exist" Oct 30 21:56:23.264298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.264248 4525 scope.go:122] "RemoveContainer" containerID="8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece" Oct 30 21:56:23.264530 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:23.264515 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece\": container with ID starting with 8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece not found: ID does not exist" containerID="8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece" Oct 30 21:56:23.264589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.264534 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece"} err="failed to get container status \"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece\": rpc error: code = NotFound desc = could not find container \"8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece\": container with ID starting with 8ef2ef2ec1b12b7acbb81a4998a21986131a5cc7d99c2b31b7fab4016cc2bece not found: ID does not exist" Oct 30 21:56:23.287791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.287742 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.287987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.287826 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.287987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.287952 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288012 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288019 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:23.288104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288056 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288088 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288099 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:23.288310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288129 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288158 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288263 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\") pod \"e7554552-a966-45ce-a189-7abd23f315db\" (UID: \"e7554552-a966-45ce-a189-7abd23f315db\") " Oct 30 21:56:23.288587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288321 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:23.288587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288338 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:23.288587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.288360 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:23.289400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.289372 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.289519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.289403 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.289519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.289418 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7554552-a966-45ce-a189-7abd23f315db-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.289519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.289432 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.289519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.289445 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.291842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.291809 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:23.291941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.291855 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "kube-api-access-nc4gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:23.291941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.291916 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:23.292084 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.292059 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo" pod "e7554552-a966-45ce-a189-7abd23f315db" (UID: "e7554552-a966-45ce-a189-7abd23f315db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:23.390845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.390759 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7554552-a966-45ce-a189-7abd23f315db-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.390845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.390791 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.390845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.390801 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4gt\" (UniqueName: \"kubernetes.io/projected/e7554552-a966-45ce-a189-7abd23f315db-kube-api-access-nc4gt\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.390845 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.390812 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7554552-a966-45ce-a189-7abd23f315db-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.401035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.401013 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:56:23.492244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.492220 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\") pod \"dd2164db-f8c6-4d60-8089-55394bb918ea\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " Oct 30 21:56:23.492453 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.492433 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\" (UniqueName: \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\") pod \"dd2164db-f8c6-4d60-8089-55394bb918ea\" (UID: \"dd2164db-f8c6-4d60-8089-55394bb918ea\") " Oct 30 21:56:23.492572 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.492551 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume" pod "dd2164db-f8c6-4d60-8089-55394bb918ea" (UID: "dd2164db-f8c6-4d60-8089-55394bb918ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:23.492920 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.492906 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2164db-f8c6-4d60-8089-55394bb918ea-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.495560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.495543 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b" pod "dd2164db-f8c6-4d60-8089-55394bb918ea" (UID: "dd2164db-f8c6-4d60-8089-55394bb918ea"). InnerVolumeSpecName "kube-api-access-cxs7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:23.593995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.593957 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxs7b\" (UniqueName: \"kubernetes.io/projected/dd2164db-f8c6-4d60-8089-55394bb918ea-kube-api-access-cxs7b\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:23.655547 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.655463 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:56:23.663705 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:23.663679 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8709fb3-openshift-default-6576d5b57c-zs6xm"] Oct 30 21:56:24.240783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.240751 4525 generic.go:356] "Generic (PLEG): container finished" podID="dd2164db-f8c6-4d60-8089-55394bb918ea" containerID="ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2" exitCode=0 Oct 30 21:56:24.241221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.240797 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" Oct 30 21:56:24.241221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.240815 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" event={"ID":"dd2164db-f8c6-4d60-8089-55394bb918ea","Type":"ContainerDied","Data":"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2"} Oct 30 21:56:24.241221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.240861 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt" event={"ID":"dd2164db-f8c6-4d60-8089-55394bb918ea","Type":"ContainerDied","Data":"91e2154971d62114b7105613973521142cbaf5327e654547ebb5ab25747889c1"} Oct 30 21:56:24.241221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.240883 4525 scope.go:122] "RemoveContainer" containerID="ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2" Oct 30 21:56:24.251474 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.251457 4525 scope.go:122] "RemoveContainer" containerID="ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2" Oct 30 21:56:24.251773 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:24.251753 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2\": container with ID starting with ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2 not found: ID does not exist" containerID="ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2" Oct 30 21:56:24.251831 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.251781 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2"} err="failed to get container status \"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2\": rpc error: code = NotFound desc = could not find container \"ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2\": container with ID starting with ba781beafc3b25b2be5c302f3609e3a45659be916493cc24d551006b5c305fa2 not found: ID does not exist" Oct 30 21:56:24.414436 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.414397 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:56:24.463950 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:24.463908 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79e10a-9676bc759-89dkt"] Oct 30 21:56:25.475622 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:25.475571 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2164db-f8c6-4d60-8089-55394bb918ea" path="/var/lib/kubelet/pods/dd2164db-f8c6-4d60-8089-55394bb918ea/volumes" Oct 30 21:56:25.476125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:25.476109 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7554552-a966-45ce-a189-7abd23f315db" path="/var/lib/kubelet/pods/e7554552-a966-45ce-a189-7abd23f315db/volumes" Oct 30 21:56:32.937233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:32.937196 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:56:35.397736 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:35.397696 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:56:36.782114 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782022 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 21:56:36.782854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782835 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.782854 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782855 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782868 4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782874 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782885 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782891 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782942 4525 cpu_manager.go:414] "containerMap: removing container" podUID="dd2164db-f8c6-4d60-8089-55394bb918ea" containerName="payload-pre-processing" Oct 30 21:56:36.783006 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.782947 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2164db-f8c6-4d60-8089-55394bb918ea" containerName="payload-pre-processing" Oct 30 21:56:36.783293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.783039 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2164db-f8c6-4d60-8089-55394bb918ea" containerName="payload-pre-processing" Oct 30 21:56:36.783293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.783049 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.783293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.783056 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf309456-53e5-4c32-a46d-4155dcc35dbb" containerName="istio-proxy" Oct 30 21:56:36.783293 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.783063 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7554552-a966-45ce-a189-7abd23f315db" containerName="istio-proxy" Oct 30 21:56:36.788088 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.788065 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.832017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.831975 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 21:56:36.836202 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.836186 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.847650 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.847626 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5bfa09-openshift-default-dockercfg-jl9rm\"" Oct 30 21:56:36.890484 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.890454 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 21:56:36.917807 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.917776 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.917987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.917815 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.917987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.917840 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.917987 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.917857 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.918132 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918102 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918213 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918339 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918320 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdrj\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918398 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918367 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918401 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918470 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.918593 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918577 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918649 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.918729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918679 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918729 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918706 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.918835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918772 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.918835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918806 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:36.918835 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918830 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.919002 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.918847 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xh9\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:36.940786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:36.940756 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 21:56:37.020020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.019977 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020032 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020129 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020178 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020221 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020203 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xh9\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020299 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020332 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020376 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020455 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020403 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020456 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020560 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020696 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdrj\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020748 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020769 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020786 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.020904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020789 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.020904 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020861 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.021165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.020944 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.021165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021005 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.021165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021031 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.021165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021046 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.021418 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021389 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.021581 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021555 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.021800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021775 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.021909 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021885 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.021977 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.021883 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.022046 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.022024 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.022566 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.022536 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.024019 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.023864 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.024674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.024650 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.025262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.025233 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.025340 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.025271 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.082248 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.082163 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xh9\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.082412 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.082285 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.083017 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.082984 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.083091 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.083040 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdrj\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\") pod \"e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.103247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.103220 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:37.152123 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.151882 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:37.553804 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.553777 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 21:56:37.554034 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:56:37.554009 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf181ba03_5400_431d_b981_937b8c1e1cc7.slice/crio-178c69b24c22ef3021285ba1e0ae90b410bb789c362edc6dec6f718a33b22d33 WatchSource:0}: Error finding container 178c69b24c22ef3021285ba1e0ae90b410bb789c362edc6dec6f718a33b22d33: Status 404 returned error can't find the container with id 178c69b24c22ef3021285ba1e0ae90b410bb789c362edc6dec6f718a33b22d33 Oct 30 21:56:37.556748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.556712 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:37.556864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.556791 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:37.556864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.556835 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:37.618495 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.618461 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 21:56:37.618995 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:56:37.618966 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460e245c_198d_4299_b681_bfb5d17a2053.slice/crio-7e1185d17ec81945191c7bc85e3389fdac404ac11b7c00686811e1c41b73dcca WatchSource:0}: Error finding container 7e1185d17ec81945191c7bc85e3389fdac404ac11b7c00686811e1c41b73dcca: Status 404 returned error can't find the container with id 7e1185d17ec81945191c7bc85e3389fdac404ac11b7c00686811e1c41b73dcca Oct 30 21:56:37.622685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.622653 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:37.622787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.622734 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:37.622787 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:37.622779 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:56:38.378300 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.378261 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" event={"ID":"460e245c-198d-4299-b681-bfb5d17a2053","Type":"ContainerStarted","Data":"801789cd191a5796a08db69d142b33245ae09b05bc4d0bead64528774f62b698"} Oct 30 21:56:38.378963 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.378936 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" event={"ID":"460e245c-198d-4299-b681-bfb5d17a2053","Type":"ContainerStarted","Data":"7e1185d17ec81945191c7bc85e3389fdac404ac11b7c00686811e1c41b73dcca"} Oct 30 21:56:38.384631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.383852 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" event={"ID":"f181ba03-5400-431d-b981-937b8c1e1cc7","Type":"ContainerStarted","Data":"a351c39453b2e1b8ddd278914ab47406b27f397d8d93fe5e20809df056c7579a"} Oct 30 21:56:38.384631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.383915 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" event={"ID":"f181ba03-5400-431d-b981-937b8c1e1cc7","Type":"ContainerStarted","Data":"178c69b24c22ef3021285ba1e0ae90b410bb789c362edc6dec6f718a33b22d33"} Oct 30 21:56:38.638694 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.636018 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podStartSLOduration=2.635996849 podStartE2EDuration="2.635996849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.582645504 +0000 UTC m=+2023.256692033" watchObservedRunningTime="2026-10-30 21:56:38.635996849 +0000 UTC m=+2023.310043385" Oct 30 21:56:38.721956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:38.721902 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podStartSLOduration=2.721885868 podStartE2EDuration="2.721885868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.720067643 +0000 UTC m=+2023.394114192" watchObservedRunningTime="2026-10-30 21:56:38.721885868 +0000 UTC m=+2023.395932404" Oct 30 21:56:39.105700 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.105128 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:39.112874 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.112835 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:39.113080 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.112891 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:56:39.162337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.162295 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:39.183635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.172743 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:56:39.183635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:39.172810 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:56:40.104591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:40.104559 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:40.105120 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:40.104649 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:56:40.153210 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:40.153162 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:56:40.153531 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:40.153492 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:56:41.103502 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:41.103466 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:41.103718 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:41.103515 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:56:41.171635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:41.154921 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:56:41.171635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:41.154998 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:56:42.103653 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:42.103591 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:42.103863 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:42.103665 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:56:42.155630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:42.155090 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:56:42.155630 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:42.155151 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:56:43.104224 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:43.104192 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:56:43.104654 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:43.104242 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:56:43.169633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:43.169567 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:56:43.169860 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:43.169654 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:56:44.108583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.108550 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:44.109069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.108868 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:44.109568 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.109552 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 21:56:44.157757 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.157729 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:44.158045 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.158023 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:44.159179 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:44.159161 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 21:56:51.731567 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:51.731532 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:56:51.838921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:51.838878 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:56:56.214276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:56.214238 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:56:56.214800 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:56.214427 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" containerID="cri-o://11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da" gracePeriod=30 Oct 30 21:56:56.732489 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:56.732419 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" podUID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" containerName="payload-processing" containerID="cri-o://474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106" gracePeriod=25 Oct 30 21:56:56.842687 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:56.842641 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" podUID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" containerName="payload-pre-processing" containerID="cri-o://11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6" gracePeriod=25 Oct 30 21:56:57.033809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.033784 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:56:57.154343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.154017 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:56:57.194211 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.194163 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\" (UniqueName: \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\") pod \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " Oct 30 21:56:57.194431 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.194407 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\") pod \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\" (UID: \"0f002bb9-dd36-4bdb-8284-78cfd2618a71\") " Oct 30 21:56:57.194778 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.194751 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume" pod "0f002bb9-dd36-4bdb-8284-78cfd2618a71" (UID: "0f002bb9-dd36-4bdb-8284-78cfd2618a71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:57.195440 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.195408 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f002bb9-dd36-4bdb-8284-78cfd2618a71-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.197007 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.196979 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp" pod "0f002bb9-dd36-4bdb-8284-78cfd2618a71" (UID: "0f002bb9-dd36-4bdb-8284-78cfd2618a71"). InnerVolumeSpecName "kube-api-access-kgdcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:57.297329 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.297240 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\" (UniqueName: \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\") pod \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " Oct 30 21:56:57.297830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.297344 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\") pod \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\" (UID: \"844df8d5-fb08-4b72-b2b5-a087c99a4f1d\") " Oct 30 21:56:57.297830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.297788 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume" pod "844df8d5-fb08-4b72-b2b5-a087c99a4f1d" (UID: "844df8d5-fb08-4b72-b2b5-a087c99a4f1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:57.298026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.298005 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.298073 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.298036 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdcp\" (UniqueName: \"kubernetes.io/projected/0f002bb9-dd36-4bdb-8284-78cfd2618a71-kube-api-access-kgdcp\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.299672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.299645 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4" pod "844df8d5-fb08-4b72-b2b5-a087c99a4f1d" (UID: "844df8d5-fb08-4b72-b2b5-a087c99a4f1d"). InnerVolumeSpecName "kube-api-access-v9pc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:57.399521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.399488 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9pc4\" (UniqueName: \"kubernetes.io/projected/844df8d5-fb08-4b72-b2b5-a087c99a4f1d-kube-api-access-v9pc4\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:57.549197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.549108 4525 generic.go:356] "Generic (PLEG): container finished" podID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" containerID="474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106" exitCode=0 Oct 30 21:56:57.549197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.549160 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" Oct 30 21:56:57.549197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.549182 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" event={"ID":"0f002bb9-dd36-4bdb-8284-78cfd2618a71","Type":"ContainerDied","Data":"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106"} Oct 30 21:56:57.549497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.549235 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk" event={"ID":"0f002bb9-dd36-4bdb-8284-78cfd2618a71","Type":"ContainerDied","Data":"cadea8b2ff3a050b3cb983e1bf34d62aef6df2e62dd1eb93966fd9bbca7bee10"} Oct 30 21:56:57.549497 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.549258 4525 scope.go:122] "RemoveContainer" containerID="474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106" Oct 30 21:56:57.550951 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.550925 4525 generic.go:356] "Generic (PLEG): container finished" podID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" containerID="11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6" exitCode=0 Oct 30 21:56:57.551086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.550982 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" Oct 30 21:56:57.551086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.550981 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" event={"ID":"844df8d5-fb08-4b72-b2b5-a087c99a4f1d","Type":"ContainerDied","Data":"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6"} Oct 30 21:56:57.551199 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.551109 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4" event={"ID":"844df8d5-fb08-4b72-b2b5-a087c99a4f1d","Type":"ContainerDied","Data":"b1d3ccffe3cb484009cd8be6081d24da06781b6da0573e5ad1e77bcd0e1d596c"} Oct 30 21:56:57.562065 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.562050 4525 scope.go:122] "RemoveContainer" containerID="474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106" Oct 30 21:56:57.562326 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:57.562306 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106\": container with ID starting with 474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106 not found: ID does not exist" containerID="474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106" Oct 30 21:56:57.562421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.562336 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106"} err="failed to get container status \"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106\": rpc error: code = NotFound desc = could not find container \"474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106\": container with ID starting with 474b129d9b68a1bb2e6c1e0861cb8651630bef19c8ea59f40ac76ea4bc7ec106 not found: ID does not exist" Oct 30 21:56:57.562421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.562358 4525 scope.go:122] "RemoveContainer" containerID="11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6" Oct 30 21:56:57.571775 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.571749 4525 scope.go:122] "RemoveContainer" containerID="11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6" Oct 30 21:56:57.572066 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:56:57.572043 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6\": container with ID starting with 11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6 not found: ID does not exist" containerID="11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6" Oct 30 21:56:57.572162 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.572071 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6"} err="failed to get container status \"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6\": rpc error: code = NotFound desc = could not find container \"11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6\": container with ID starting with 11dba06159036f55438bc8bea8d2286246951b873c90ef135a5d457862086dc6 not found: ID does not exist" Oct 30 21:56:57.715967 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.715928 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:56:57.765897 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.765864 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-259e81-66fd4765f5-zxdm4"] Oct 30 21:56:57.829178 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.829101 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:56:57.833670 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:57.833640 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-259e81-f7587d664-wgsrk"] Oct 30 21:56:58.975391 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:58.975358 4525 patch_prober.go:29] interesting pod/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:58.975812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:58.975419 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:59.140730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.140693 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:56:59.140933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.140912 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" containerID="cri-o://174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c" gracePeriod=30 Oct 30 21:56:59.168758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.168725 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:56:59.168986 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.168930 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" containerID="cri-o://59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d" gracePeriod=30 Oct 30 21:56:59.475577 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.475537 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" path="/var/lib/kubelet/pods/0f002bb9-dd36-4bdb-8284-78cfd2618a71/volumes" Oct 30 21:56:59.476076 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:56:59.476061 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" path="/var/lib/kubelet/pods/844df8d5-fb08-4b72-b2b5-a087c99a4f1d/volumes" Oct 30 21:57:00.915994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:00.915960 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:00.916491 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:00.916029 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:01.481994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.481930 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:57:01.561378 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561334 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561512 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561578 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561566 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561593 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561700 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561748 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561856 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561824 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.561957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561882 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.561957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561919 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.562064 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.561962 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.562206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.562180 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.562331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.562273 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\") pod \"84218860-63ed-4fe2-9489-41bf1bd3dafc\" (UID: \"84218860-63ed-4fe2-9489-41bf1bd3dafc\") " Oct 30 21:57:01.562331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.562274 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:01.562575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.562551 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.562957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.562937 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.563106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.563084 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.563203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.563111 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.563203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.563128 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84218860-63ed-4fe2-9489-41bf1bd3dafc-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.563203 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.563143 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.564166 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.564138 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "kube-api-access-gd65f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.564320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.564301 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:01.564382 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.564344 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:01.564546 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.564521 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy" pod "84218860-63ed-4fe2-9489-41bf1bd3dafc" (UID: "84218860-63ed-4fe2-9489-41bf1bd3dafc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:01.586447 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.586415 4525 generic.go:356] "Generic (PLEG): container finished" podID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerID="11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da" exitCode=0 Oct 30 21:57:01.586598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.586484 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" Oct 30 21:57:01.586598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.586504 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" event={"ID":"84218860-63ed-4fe2-9489-41bf1bd3dafc","Type":"ContainerDied","Data":"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da"} Oct 30 21:57:01.586598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.586561 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4" event={"ID":"84218860-63ed-4fe2-9489-41bf1bd3dafc","Type":"ContainerDied","Data":"22dfe60d5746fb966dc4c027b2db64d3fae16c2ae3f2b322285208c0c77a4f07"} Oct 30 21:57:01.586598 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.586582 4525 scope.go:122] "RemoveContainer" containerID="11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da" Oct 30 21:57:01.598941 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.598916 4525 scope.go:122] "RemoveContainer" containerID="11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da" Oct 30 21:57:01.599294 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:01.599275 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da\": container with ID starting with 11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da not found: ID does not exist" containerID="11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da" Oct 30 21:57:01.599349 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.599305 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da"} err="failed to get container status \"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da\": rpc error: code = NotFound desc = could not find container \"11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da\": container with ID starting with 11218986ec6becc763dd01c721328588423956b180755c77e4c7278c3b0567da not found: ID does not exist" Oct 30 21:57:01.665231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.665194 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.665231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.665224 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd65f\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-kube-api-access-gd65f\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.665231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.665234 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.665486 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.665243 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84218860-63ed-4fe2-9489-41bf1bd3dafc-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:01.781864 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.781687 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:57:01.785358 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.785325 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d68156-openshift-default-797fb546b7-hw2f4"] Oct 30 21:57:01.948142 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.948102 4525 patch_prober.go:29] interesting pod/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:01.948580 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:01.948169 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:03.475163 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:03.475135 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" path="/var/lib/kubelet/pods/84218860-63ed-4fe2-9489-41bf1bd3dafc/volumes" Oct 30 21:57:04.482781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.482753 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:57:04.486390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.486369 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:57:04.606842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.606809 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.606922 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.606966 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.606999 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607037 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607253 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607071 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607253 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607152 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607253 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607188 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607370 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607277 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607371 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607419 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607406 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607402 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.607490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607449 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607495 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607535 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.607642 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607636 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607669 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.607786 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.607669 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608071 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608249 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608420 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\") pod \"dc986014-3bca-4681-bc9d-ddec6fe08e48\" (UID: \"dc986014-3bca-4681-bc9d-ddec6fe08e48\") " Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608486 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\") pod \"04a81afe-4ebc-47bf-90c5-4039f4646392\" (UID: \"04a81afe-4ebc-47bf-90c5-4039f4646392\") " Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608669 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.608958 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609435 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609459 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609501 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609536 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609583 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609620 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609691 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a81afe-4ebc-47bf-90c5-4039f4646392-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.609992 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "kube-api-access-2v47g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.610295 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.610447 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.610484 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.610677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.610648 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:04.612003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.611952 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.612241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.612020 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:04.612241 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.612211 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:04.612781 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.612758 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token" pod "dc986014-3bca-4681-bc9d-ddec6fe08e48" (UID: "dc986014-3bca-4681-bc9d-ddec6fe08e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.613217 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.613195 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:04.613733 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.613708 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6" pod "04a81afe-4ebc-47bf-90c5-4039f4646392" (UID: "04a81afe-4ebc-47bf-90c5-4039f4646392"). InnerVolumeSpecName "kube-api-access-cppq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:04.622463 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.622428 4525 generic.go:356] "Generic (PLEG): container finished" podID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerID="59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d" exitCode=0 Oct 30 21:57:04.622644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.622534 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" Oct 30 21:57:04.622644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.622571 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" event={"ID":"04a81afe-4ebc-47bf-90c5-4039f4646392","Type":"ContainerDied","Data":"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d"} Oct 30 21:57:04.622776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.622652 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25" event={"ID":"04a81afe-4ebc-47bf-90c5-4039f4646392","Type":"ContainerDied","Data":"ef78dd3eb9dfb19dd5a24150770b126d58a71495b0bc3883ca41f8d10e6a032f"} Oct 30 21:57:04.622776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.622684 4525 scope.go:122] "RemoveContainer" containerID="59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d" Oct 30 21:57:04.624776 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.624748 4525 generic.go:356] "Generic (PLEG): container finished" podID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerID="174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c" exitCode=0 Oct 30 21:57:04.624883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.624821 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" Oct 30 21:57:04.624883 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.624841 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" event={"ID":"dc986014-3bca-4681-bc9d-ddec6fe08e48","Type":"ContainerDied","Data":"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c"} Oct 30 21:57:04.624995 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.624884 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp" event={"ID":"dc986014-3bca-4681-bc9d-ddec6fe08e48","Type":"ContainerDied","Data":"f1e5a50c227e39be53c2d8aa8e55237de18a2a888d9dfde2b8a93712845ba04e"} Oct 30 21:57:04.640026 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.640007 4525 scope.go:122] "RemoveContainer" containerID="59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d" Oct 30 21:57:04.640346 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:04.640322 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d\": container with ID starting with 59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d not found: ID does not exist" containerID="59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d" Oct 30 21:57:04.640400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.640356 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d"} err="failed to get container status \"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d\": rpc error: code = NotFound desc = could not find container \"59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d\": container with ID starting with 59dcd9ef664610b426a2b14ebe652a5db1e5a8eafd079c6f5d6cb9fe3b24b67d not found: ID does not exist" Oct 30 21:57:04.640400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.640373 4525 scope.go:122] "RemoveContainer" containerID="174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c" Oct 30 21:57:04.650181 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.650162 4525 scope.go:122] "RemoveContainer" containerID="174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c" Oct 30 21:57:04.650462 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:04.650443 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c\": container with ID starting with 174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c not found: ID does not exist" containerID="174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c" Oct 30 21:57:04.650523 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.650469 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c"} err="failed to get container status \"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c\": rpc error: code = NotFound desc = could not find container \"174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c\": container with ID starting with 174dc200e957fe0e86c7859a2dc223110eb893fc094ff919151a71b7ed93d67c not found: ID does not exist" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711372 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711397 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711408 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cppq6\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-kube-api-access-cppq6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711418 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711426 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711443 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711438 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v47g\" (UniqueName: \"kubernetes.io/projected/dc986014-3bca-4681-bc9d-ddec6fe08e48-kube-api-access-2v47g\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711452 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711464 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc986014-3bca-4681-bc9d-ddec6fe08e48-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711473 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711483 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a81afe-4ebc-47bf-90c5-4039f4646392-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711491 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc986014-3bca-4681-bc9d-ddec6fe08e48-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.711771 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.711499 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc986014-3bca-4681-bc9d-ddec6fe08e48-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:04.780969 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.780936 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:57:04.784364 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.784337 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-2tnmp"] Oct 30 21:57:04.966200 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.966124 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:57:04.978355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:04.978316 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-r4s25"] Oct 30 21:57:05.396516 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:05.396478 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:57:05.475882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:05.475849 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" path="/var/lib/kubelet/pods/04a81afe-4ebc-47bf-90c5-4039f4646392/volumes" Oct 30 21:57:05.476405 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:05.476389 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" path="/var/lib/kubelet/pods/dc986014-3bca-4681-bc9d-ddec6fe08e48/volumes" Oct 30 21:57:07.457640 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:07.457586 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:57:07.458044 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:07.457845 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" containerID="cri-o://e2b34db45369dd29084539de63de3b602e69578e58da6f99f6b07c83bf38e20f" gracePeriod=30 Oct 30 21:57:12.686526 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.686484 4525 generic.go:356] "Generic (PLEG): container finished" podID="ae48d365-ae0d-45ff-9b07-761218945f63" containerID="e2b34db45369dd29084539de63de3b602e69578e58da6f99f6b07c83bf38e20f" exitCode=0 Oct 30 21:57:12.687087 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.686536 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" event={"ID":"ae48d365-ae0d-45ff-9b07-761218945f63","Type":"ContainerDied","Data":"e2b34db45369dd29084539de63de3b602e69578e58da6f99f6b07c83bf38e20f"} Oct 30 21:57:12.738195 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.738137 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:57:12.746264 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746241 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746287 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746386 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746368 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746418 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746472 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746517 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746579 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746472 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.746671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746653 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746740 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746673 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.746795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746733 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746780 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\") pod \"ae48d365-ae0d-45ff-9b07-761218945f63\" (UID: \"ae48d365-ae0d-45ff-9b07-761218945f63\") " Oct 30 21:57:12.746908 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.746870 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.747038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747016 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.747193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747174 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:12.747505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747485 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.747584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747510 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.747584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747526 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.747584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747540 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae48d365-ae0d-45ff-9b07-761218945f63-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.747584 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.747553 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.748939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.748914 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:12.748939 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.748926 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "kube-api-access-6wctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:12.749320 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.749296 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:12.749418 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.749323 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token" pod "ae48d365-ae0d-45ff-9b07-761218945f63" (UID: "ae48d365-ae0d-45ff-9b07-761218945f63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:12.849901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.849870 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae48d365-ae0d-45ff-9b07-761218945f63-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.849901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.849899 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.849901 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.849909 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae48d365-ae0d-45ff-9b07-761218945f63-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:12.850165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:12.849920 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wctf\" (UniqueName: \"kubernetes.io/projected/ae48d365-ae0d-45ff-9b07-761218945f63-kube-api-access-6wctf\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:13.695384 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:13.695346 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" event={"ID":"ae48d365-ae0d-45ff-9b07-761218945f63","Type":"ContainerDied","Data":"1dd1eda9595bec43872758e9e23ee370e18148f38716471e6c39053a740c1f43"} Oct 30 21:57:13.695886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:13.695410 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj" Oct 30 21:57:13.695886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:13.695417 4525 scope.go:122] "RemoveContainer" containerID="e2b34db45369dd29084539de63de3b602e69578e58da6f99f6b07c83bf38e20f" Oct 30 21:57:13.912331 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:13.912156 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:57:13.915870 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:13.915841 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-h9grj"] Oct 30 21:57:15.474790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:15.474755 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" path="/var/lib/kubelet/pods/ae48d365-ae0d-45ff-9b07-761218945f63/volumes" Oct 30 21:57:15.761809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:15.761721 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:57:15.855843 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:15.855803 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:57:20.762707 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:20.762646 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" podUID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" containerName="payload-processing" containerID="cri-o://b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760" gracePeriod=25 Oct 30 21:57:20.856222 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:20.856154 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" podUID="fec442a6-643a-4837-b8e5-3bac41636426" containerName="payload-pre-processing" containerID="cri-o://46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d" gracePeriod=25 Oct 30 21:57:21.148068 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.148037 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:57:21.180673 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.180646 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:57:21.240735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.240691 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\" (UniqueName: \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\") pod \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " Oct 30 21:57:21.240735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.240741 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\") pod \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\" (UID: \"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa\") " Oct 30 21:57:21.241131 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.241105 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume" pod "c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" (UID: "c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:21.241232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.241219 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.243085 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.243056 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4" pod "c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" (UID: "c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa"). InnerVolumeSpecName "kube-api-access-jxgs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:21.341696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.341573 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\") pod \"fec442a6-643a-4837-b8e5-3bac41636426\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " Oct 30 21:57:21.341839 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.341828 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\" (UniqueName: \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\") pod \"fec442a6-643a-4837-b8e5-3bac41636426\" (UID: \"fec442a6-643a-4837-b8e5-3bac41636426\") " Oct 30 21:57:21.341960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.341930 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume" pod "fec442a6-643a-4837-b8e5-3bac41636426" (UID: "fec442a6-643a-4837-b8e5-3bac41636426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:21.342242 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.342224 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxgs4\" (UniqueName: \"kubernetes.io/projected/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa-kube-api-access-jxgs4\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.342306 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.342244 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec442a6-643a-4837-b8e5-3bac41636426-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.344098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.344075 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9" pod "fec442a6-643a-4837-b8e5-3bac41636426" (UID: "fec442a6-643a-4837-b8e5-3bac41636426"). InnerVolumeSpecName "kube-api-access-rnwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:21.443385 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.443345 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwk9\" (UniqueName: \"kubernetes.io/projected/fec442a6-643a-4837-b8e5-3bac41636426-kube-api-access-rnwk9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.754165 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.754132 4525 generic.go:356] "Generic (PLEG): container finished" podID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" containerID="b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760" exitCode=0 Oct 30 21:57:21.754346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.754180 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" Oct 30 21:57:21.754346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.754212 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" event={"ID":"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa","Type":"ContainerDied","Data":"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760"} Oct 30 21:57:21.754346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.754267 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq" event={"ID":"c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa","Type":"ContainerDied","Data":"8278e7f839c46476d5972a9f3a4e22609cce0f718fa759a2e4415561c756e04f"} Oct 30 21:57:21.754346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.754297 4525 scope.go:122] "RemoveContainer" containerID="b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760" Oct 30 21:57:21.756086 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.756060 4525 generic.go:356] "Generic (PLEG): container finished" podID="fec442a6-643a-4837-b8e5-3bac41636426" containerID="46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d" exitCode=0 Oct 30 21:57:21.756219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.756118 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" Oct 30 21:57:21.756219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.756204 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" event={"ID":"fec442a6-643a-4837-b8e5-3bac41636426","Type":"ContainerDied","Data":"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d"} Oct 30 21:57:21.756312 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.756240 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd" event={"ID":"fec442a6-643a-4837-b8e5-3bac41636426","Type":"ContainerDied","Data":"4c52e483270b19777245a3bd12be3e0ae99cd644bbcc9b1636f54d8fc0d9fcca"} Oct 30 21:57:21.768888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.768112 4525 scope.go:122] "RemoveContainer" containerID="b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760" Oct 30 21:57:21.768888 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:21.768493 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760\": container with ID starting with b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760 not found: ID does not exist" containerID="b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760" Oct 30 21:57:21.768888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.768527 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760"} err="failed to get container status \"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760\": rpc error: code = NotFound desc = could not find container \"b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760\": container with ID starting with b3c37f4c0b3a8af6d7891255ee5224a7806a003ea2ed15f24777745538f28760 not found: ID does not exist" Oct 30 21:57:21.768888 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.768549 4525 scope.go:122] "RemoveContainer" containerID="46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d" Oct 30 21:57:21.787164 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.787140 4525 scope.go:122] "RemoveContainer" containerID="46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d" Oct 30 21:57:21.787499 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:21.787478 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d\": container with ID starting with 46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d not found: ID does not exist" containerID="46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d" Oct 30 21:57:21.787576 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.787507 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d"} err="failed to get container status \"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d\": rpc error: code = NotFound desc = could not find container \"46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d\": container with ID starting with 46d5a451aa479239c04da39d186f283cbabd1c74c9ccda3dcac7e3574fcc276d not found: ID does not exist" Oct 30 21:57:21.950264 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.950234 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:57:21.954929 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:21.954881 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e51c52-5868497b48-v7mwq"] Oct 30 21:57:22.061880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:22.061784 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:57:22.070765 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:22.070727 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e51c52-54bf949b87-5z9gd"] Oct 30 21:57:23.483719 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:23.483688 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" path="/var/lib/kubelet/pods/c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa/volumes" Oct 30 21:57:23.484125 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:23.484112 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec442a6-643a-4837-b8e5-3bac41636426" path="/var/lib/kubelet/pods/fec442a6-643a-4837-b8e5-3bac41636426/volumes" Oct 30 21:57:35.227351 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.227304 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:57:35.261632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.227575 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" containerID="cri-o://fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e" gracePeriod=30 Oct 30 21:57:35.415651 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:35.412554 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:57:35.687886 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.687861 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:57:35.759124 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.759077 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\" (UniqueName: \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\") pod \"992e26f2-761b-4cf6-8a52-936db77eb417\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " Oct 30 21:57:35.759341 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.759195 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\") pod \"992e26f2-761b-4cf6-8a52-936db77eb417\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " Oct 30 21:57:35.759407 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.759373 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\") pod \"992e26f2-761b-4cf6-8a52-936db77eb417\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " Oct 30 21:57:35.759490 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.759474 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\") pod \"992e26f2-761b-4cf6-8a52-936db77eb417\" (UID: \"992e26f2-761b-4cf6-8a52-936db77eb417\") " Oct 30 21:57:35.762983 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.760082 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca" pod "992e26f2-761b-4cf6-8a52-936db77eb417" (UID: "992e26f2-761b-4cf6-8a52-936db77eb417"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:35.762983 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.761446 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/992e26f2-761b-4cf6-8a52-936db77eb417-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.772477 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.772407 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls" pod "992e26f2-761b-4cf6-8a52-936db77eb417" (UID: "992e26f2-761b-4cf6-8a52-936db77eb417"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:35.772858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.772831 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb" pod "992e26f2-761b-4cf6-8a52-936db77eb417" (UID: "992e26f2-761b-4cf6-8a52-936db77eb417"). InnerVolumeSpecName "kube-api-access-95rfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:35.776737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.776704 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs" pod "992e26f2-761b-4cf6-8a52-936db77eb417" (UID: "992e26f2-761b-4cf6-8a52-936db77eb417"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:35.863215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.863111 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.863215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.863140 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rfb\" (UniqueName: \"kubernetes.io/projected/992e26f2-761b-4cf6-8a52-936db77eb417-kube-api-access-95rfb\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.863215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.863152 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/992e26f2-761b-4cf6-8a52-936db77eb417-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.865154 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.865127 4525 generic.go:356] "Generic (PLEG): container finished" podID="992e26f2-761b-4cf6-8a52-936db77eb417" containerID="fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e" exitCode=0 Oct 30 21:57:35.865286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.865192 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerDied","Data":"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e"} Oct 30 21:57:35.865286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.865194 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" Oct 30 21:57:35.865286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.865233 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh" event={"ID":"992e26f2-761b-4cf6-8a52-936db77eb417","Type":"ContainerDied","Data":"02b5a727c5fb054b292ef121805768db43091e615810f6032e1f98c64dbf9c43"} Oct 30 21:57:35.865286 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.865256 4525 scope.go:122] "RemoveContainer" containerID="fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e" Oct 30 21:57:35.876020 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.875996 4525 scope.go:122] "RemoveContainer" containerID="c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624" Oct 30 21:57:35.885574 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.885548 4525 scope.go:122] "RemoveContainer" containerID="fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e" Oct 30 21:57:35.885887 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:35.885866 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e\": container with ID starting with fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e not found: ID does not exist" containerID="fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e" Oct 30 21:57:35.885968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.885894 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e"} err="failed to get container status \"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e\": rpc error: code = NotFound desc = could not find container \"fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e\": container with ID starting with fa731dbb94e4b0378b0eb55d32ea0f89bd0d21ee1d9ea13190fbb24253deda3e not found: ID does not exist" Oct 30 21:57:35.885968 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.885916 4525 scope.go:122] "RemoveContainer" containerID="c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624" Oct 30 21:57:35.886205 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:57:35.886186 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624\": container with ID starting with c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624 not found: ID does not exist" containerID="c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624" Oct 30 21:57:35.886252 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:35.886211 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624"} err="failed to get container status \"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624\": rpc error: code = NotFound desc = could not find container \"c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624\": container with ID starting with c1366bb45dbb69a1961bc6a821fbcdba9cd29634f99208cfc9876d455c42b624 not found: ID does not exist" Oct 30 21:57:36.033511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:36.033475 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:57:36.037317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:36.037282 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-79e10a-d4cb65d68-5dhsh"] Oct 30 21:57:37.479061 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.479018 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" path="/var/lib/kubelet/pods/992e26f2-761b-4cf6-8a52-936db77eb417/volumes" Oct 30 21:57:37.479488 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.479470 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 21:57:37.480362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480340 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fec442a6-643a-4837-b8e5-3bac41636426" containerName="payload-pre-processing" Oct 30 21:57:37.480429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480366 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec442a6-643a-4837-b8e5-3bac41636426" containerName="payload-pre-processing" Oct 30 21:57:37.480429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480391 4525 cpu_manager.go:414] "containerMap: removing container" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.480429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480400 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.480429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480412 4525 cpu_manager.go:414] "containerMap: removing container" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" Oct 30 21:57:37.480429 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480419 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480456 4525 cpu_manager.go:414] "containerMap: removing container" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480464 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480485 4525 cpu_manager.go:414] "containerMap: removing container" podUID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" containerName="payload-pre-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480492 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" containerName="payload-pre-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480518 4525 cpu_manager.go:414] "containerMap: removing container" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480525 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480542 4525 cpu_manager.go:414] "containerMap: removing container" podUID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" containerName="payload-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480549 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" containerName="payload-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480569 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" containerName="payload-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480576 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" containerName="payload-processing" Oct 30 21:57:37.480627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480592 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" Oct 30 21:57:37.481004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480600 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" Oct 30 21:57:37.481004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480973 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f002bb9-dd36-4bdb-8284-78cfd2618a71" containerName="payload-processing" Oct 30 21:57:37.481004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480987 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae48d365-ae0d-45ff-9b07-761218945f63" containerName="istio-proxy" Oct 30 21:57:37.481004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.480998 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481009 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec442a6-643a-4837-b8e5-3bac41636426" containerName="payload-pre-processing" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481020 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481031 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc986014-3bca-4681-bc9d-ddec6fe08e48" containerName="istio-proxy" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481044 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="84218860-63ed-4fe2-9489-41bf1bd3dafc" containerName="istio-proxy" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481054 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="844df8d5-fb08-4b72-b2b5-a087c99a4f1d" containerName="payload-pre-processing" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481064 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a81afe-4ebc-47bf-90c5-4039f4646392" containerName="istio-proxy" Oct 30 21:57:37.481136 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.481073 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31f4e0a-75d2-4f1e-85b9-089d78b6a2fa" containerName="payload-processing" Oct 30 21:57:37.486260 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.486238 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.523743 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.523704 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 21:57:37.524537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.524512 4525 cpu_manager.go:414] "containerMap: removing container" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.524537 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.524535 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992e26f2-761b-4cf6-8a52-936db77eb417" containerName="maas-api" Oct 30 21:57:37.528223 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.528201 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 21:57:37.528338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.528322 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.532575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.532551 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.546324 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.546299 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 21:57:37.569643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.569598 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lfxgk\"" Oct 30 21:57:37.590421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590293 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6dz\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.590421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590358 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.590421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590389 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrb2\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590418 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590485 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590515 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590553 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590579 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590618 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590682 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590707 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.590777 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590740 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590794 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590843 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590896 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.590932 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhg4s\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591000 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591057 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591107 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.591191 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591137 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591221 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591255 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591291 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591374 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591436 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591479 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.594219 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.591519 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.599553 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.599527 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 21:57:37.604346 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.604320 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 21:57:37.692913 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.692874 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.692943 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.692989 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693023 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693055 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693083 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6dz\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.693133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693123 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693145 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrb2\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693168 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693213 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693241 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693281 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693302 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693326 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693370 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693385 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693395 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693394 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.693454 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693444 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693469 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693529 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693580 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693638 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693672 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhg4s\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693708 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693743 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693774 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693822 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693852 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693931 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693965 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.694288 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.693972 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.695055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.694692 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.695055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.694773 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.695055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.695003 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.695259 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.695237 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.695474 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.695449 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.696059 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696033 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.696193 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696168 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.696373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696334 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.696512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696384 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.696512 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696454 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.696664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696560 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.696664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696562 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.696664 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.696628 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.697063 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.697041 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.697671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.697600 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.698830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.698812 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.755348 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.755239 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6dz\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.756308 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.756252 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xrbg8\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:37.756464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.756347 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrb2\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.756591 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.756573 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mrk7h\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.756735 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.756697 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhg4s\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.757444 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.757423 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jlp79\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.800861 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.800829 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:37.844048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.844016 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:37.851528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:37.851501 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:38.238394 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.238370 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 21:57:38.239013 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:57:38.238989 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea6b3ee_cbbf_44af_826e_647792e2cb7c.slice/crio-276f97cf55261c8904c169a0d4667ac0c1d7788697af005679f3be2fe055175e WatchSource:0}: Error finding container 276f97cf55261c8904c169a0d4667ac0c1d7788697af005679f3be2fe055175e: Status 404 returned error can't find the container with id 276f97cf55261c8904c169a0d4667ac0c1d7788697af005679f3be2fe055175e Oct 30 21:57:38.241730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.241694 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.241817 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.241762 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.241817 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.241791 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.292030 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.292003 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 21:57:38.296896 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:57:38.296870 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39db119_794c_469d_a941_8f28110bd3bd.slice/crio-bde517620206a9239be777867b4c5b9717d9075b52b84d2753f3d4acd204da60 WatchSource:0}: Error finding container bde517620206a9239be777867b4c5b9717d9075b52b84d2753f3d4acd204da60: Status 404 returned error can't find the container with id bde517620206a9239be777867b4c5b9717d9075b52b84d2753f3d4acd204da60 Oct 30 21:57:38.299842 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.299808 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.299943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.299888 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.299943 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.299919 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.354192 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.354158 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 21:57:38.357132 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:57:38.357103 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9bcc39_b9df_4cb4_880a_3fab5a628cb7.slice/crio-04029fb5d38d5a3d0ef6e05fa85475429804fe27463438ed0fbef3c7a6ab1b23 WatchSource:0}: Error finding container 04029fb5d38d5a3d0ef6e05fa85475429804fe27463438ed0fbef3c7a6ab1b23: Status 404 returned error can't find the container with id 04029fb5d38d5a3d0ef6e05fa85475429804fe27463438ed0fbef3c7a6ab1b23 Oct 30 21:57:38.360748 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.360710 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.360840 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.360797 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.360880 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.360840 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:38.877635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.872099 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:57:38.941039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.940385 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" event={"ID":"be9bcc39-b9df-4cb4-880a-3fab5a628cb7","Type":"ContainerStarted","Data":"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a"} Oct 30 21:57:38.941039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.940517 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" event={"ID":"be9bcc39-b9df-4cb4-880a-3fab5a628cb7","Type":"ContainerStarted","Data":"04029fb5d38d5a3d0ef6e05fa85475429804fe27463438ed0fbef3c7a6ab1b23"} Oct 30 21:57:38.975238 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.975198 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" event={"ID":"e39db119-794c-469d-a941-8f28110bd3bd","Type":"ContainerStarted","Data":"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5"} Oct 30 21:57:38.976298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.976271 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" event={"ID":"e39db119-794c-469d-a941-8f28110bd3bd","Type":"ContainerStarted","Data":"bde517620206a9239be777867b4c5b9717d9075b52b84d2753f3d4acd204da60"} Oct 30 21:57:38.978078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.978054 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" event={"ID":"4ea6b3ee-cbbf-44af-826e-647792e2cb7c","Type":"ContainerStarted","Data":"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172"} Oct 30 21:57:38.978258 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:38.978246 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" event={"ID":"4ea6b3ee-cbbf-44af-826e-647792e2cb7c","Type":"ContainerStarted","Data":"276f97cf55261c8904c169a0d4667ac0c1d7788697af005679f3be2fe055175e"} Oct 30 21:57:39.232220 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.232100 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podStartSLOduration=2.232075592 podStartE2EDuration="2.232075592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:39.158418106 +0000 UTC m=+2083.832464653" watchObservedRunningTime="2026-10-30 21:57:39.232075592 +0000 UTC m=+2083.906122128" Oct 30 21:57:39.305632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.300402 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podStartSLOduration=2.300363365 podStartE2EDuration="2.300363365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:39.286371978 +0000 UTC m=+2083.960418541" watchObservedRunningTime="2026-10-30 21:57:39.300363365 +0000 UTC m=+2083.974409903" Oct 30 21:57:39.815634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.814745 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:39.816018 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.815987 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:39.816231 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.816203 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:39.847460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.847426 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:39.847865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.847838 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:39.865634 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.848018 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:39.868236 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.866127 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:39.868589 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.868514 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:39.868789 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:39.868765 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:40.802375 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.802340 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:40.803048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.803000 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:40.847791 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.847736 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:40.848082 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.848056 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:40.851986 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.851954 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:40.852222 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:40.852190 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:41.802635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.802178 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:41.802635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.802239 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:41.847636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.844630 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:41.847636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.844691 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:41.859645 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.857052 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:41.859645 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:41.857115 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:42.815243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.815208 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.815923 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.815891 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:42.845633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.845117 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.845633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.845196 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:42.865632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.855128 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.865632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:42.855223 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:43.807703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.807218 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.807703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.807301 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:43.847633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.844330 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.847633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.844407 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:43.859635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.858352 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.859635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.858413 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:43.877199 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:43.875676 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" podUID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" containerName="payload-pre-processing" containerID="cri-o://7936ecc5765461e14c3ce400148dc5185eba37da946ce19a2d99f7fd3b892568" gracePeriod=25 Oct 30 21:57:44.120027 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.119989 4525 generic.go:356] "Generic (PLEG): container finished" podID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" containerID="7936ecc5765461e14c3ce400148dc5185eba37da946ce19a2d99f7fd3b892568" exitCode=0 Oct 30 21:57:44.120255 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.120061 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" event={"ID":"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd","Type":"ContainerDied","Data":"7936ecc5765461e14c3ce400148dc5185eba37da946ce19a2d99f7fd3b892568"} Oct 30 21:57:44.618643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.618585 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:57:44.658633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.638094 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\" (UniqueName: \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\") pod \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " Oct 30 21:57:44.658633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.638221 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\") pod \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\" (UID: \"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd\") " Oct 30 21:57:44.695983 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.690249 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j" pod "cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" (UID: "cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd"). InnerVolumeSpecName "kube-api-access-scj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:44.695983 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.691871 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume" pod "cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" (UID: "cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:44.742003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.741961 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:44.742262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.742245 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scj5j\" (UniqueName: \"kubernetes.io/projected/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd-kube-api-access-scj5j\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:44.816685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.802383 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:44.816685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.802481 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:44.816685 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.807164 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podStartSLOduration=7.807146174 podStartE2EDuration="7.807146174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:39.658874362 +0000 UTC m=+2084.332920899" watchObservedRunningTime="2026-10-30 21:57:44.807146174 +0000 UTC m=+2089.481192712" Oct 30 21:57:44.859445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.853040 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:44.859445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.853104 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:44.859445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.854512 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:44.859445 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:44.854565 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:45.189051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.189019 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" event={"ID":"cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd","Type":"ContainerDied","Data":"a1d1bd60b9de46e2c79ee520a544fdcb740f09041c99663194ff4db81bf3159d"} Oct 30 21:57:45.189345 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.189326 4525 scope.go:122] "RemoveContainer" containerID="7936ecc5765461e14c3ce400148dc5185eba37da946ce19a2d99f7fd3b892568" Oct 30 21:57:45.189631 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.189592 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6" Oct 30 21:57:45.394465 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.394423 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:57:45.402624 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.402559 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c92783-669dbc4b9f-zj6l6"] Oct 30 21:57:45.478013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.477926 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" path="/var/lib/kubelet/pods/cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd/volumes" Oct 30 21:57:45.801343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.801251 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:45.801343 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.801325 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:45.845269 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.845238 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:45.845470 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.845297 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:45.852737 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.852703 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:45.852859 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:45.852766 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:46.801833 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.801792 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 21:57:46.802321 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.801857 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 21:57:46.844722 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.844664 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:57:46.844933 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.844763 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:57:46.852229 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.852196 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:57:46.852374 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:46.852253 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 21:57:48.372659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.363659 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:48.372659 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.365763 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:48.374028 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.373950 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 21:57:48.515922 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.515887 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:48.516367 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.516339 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:48.517261 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.517242 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 21:57:48.620315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.620281 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:48.621031 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.621002 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:57:48.621703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:57:48.621683 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 21:58:05.432981 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:05.432942 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:58:05.667262 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:05.667225 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:58:15.478836 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.478765 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:58:15.479279 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.478933 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" podUID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" containerName="maas-api" containerID="cri-o://91196bd860c9a395592eeaffead0447644d67c181994e2c7b94897b7df16b0bf" gracePeriod=30 Oct 30 21:58:15.675371 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.675337 4525 generic.go:356] "Generic (PLEG): container finished" podID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" containerID="91196bd860c9a395592eeaffead0447644d67c181994e2c7b94897b7df16b0bf" exitCode=0 Oct 30 21:58:15.675586 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.675477 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" event={"ID":"a1b931a0-fa52-4f2a-a309-b21c939a70d7","Type":"ContainerDied","Data":"91196bd860c9a395592eeaffead0447644d67c181994e2c7b94897b7df16b0bf"} Oct 30 21:58:15.816386 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.816359 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:58:15.933703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.932198 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\") pod \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " Oct 30 21:58:15.933703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.932290 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\" (UniqueName: \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\") pod \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " Oct 30 21:58:15.933703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.932424 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\") pod \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " Oct 30 21:58:15.933703 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.932620 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\") pod \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\" (UID: \"a1b931a0-fa52-4f2a-a309-b21c939a70d7\") " Oct 30 21:58:15.947284 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.942142 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls" pod "a1b931a0-fa52-4f2a-a309-b21c939a70d7" (UID: "a1b931a0-fa52-4f2a-a309-b21c939a70d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:15.952971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.952936 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca" pod "a1b931a0-fa52-4f2a-a309-b21c939a70d7" (UID: "a1b931a0-fa52-4f2a-a309-b21c939a70d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:15.968373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.968315 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs" pod "a1b931a0-fa52-4f2a-a309-b21c939a70d7" (UID: "a1b931a0-fa52-4f2a-a309-b21c939a70d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:15.969078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:15.969037 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t" pod "a1b931a0-fa52-4f2a-a309-b21c939a70d7" (UID: "a1b931a0-fa52-4f2a-a309-b21c939a70d7"). InnerVolumeSpecName "kube-api-access-k6h4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:16.034519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.034487 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b931a0-fa52-4f2a-a309-b21c939a70d7-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.034519 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.034518 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6h4t\" (UniqueName: \"kubernetes.io/projected/a1b931a0-fa52-4f2a-a309-b21c939a70d7-kube-api-access-k6h4t\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.034758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.034533 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.034758 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.034546 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b931a0-fa52-4f2a-a309-b21c939a70d7-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.683327 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.683297 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" Oct 30 21:58:16.683839 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.683323 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk" event={"ID":"a1b931a0-fa52-4f2a-a309-b21c939a70d7","Type":"ContainerDied","Data":"2e1e2708929e4311e1c9eaab802e6039bed5aec75e246a754403abf1375dc96b"} Oct 30 21:58:16.683839 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.683403 4525 scope.go:122] "RemoveContainer" containerID="91196bd860c9a395592eeaffead0447644d67c181994e2c7b94897b7df16b0bf" Oct 30 21:58:16.852680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.852636 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:58:16.902618 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:16.902580 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-xtvzk"] Oct 30 21:58:17.476196 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:17.476162 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" path="/var/lib/kubelet/pods/a1b931a0-fa52-4f2a-a309-b21c939a70d7/volumes" Oct 30 21:58:30.844752 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:30.844716 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:58:32.475677 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.475641 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:58:32.477571 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.477538 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" podUID="3be7072c-9090-4137-a303-45d5f2e3191d" containerName="maas-api" containerID="cri-o://fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400" gracePeriod=30 Oct 30 21:58:32.817647 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.812558 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:58:32.834599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.828410 4525 generic.go:356] "Generic (PLEG): container finished" podID="3be7072c-9090-4137-a303-45d5f2e3191d" containerID="fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400" exitCode=0 Oct 30 21:58:32.834599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.828525 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" event={"ID":"3be7072c-9090-4137-a303-45d5f2e3191d","Type":"ContainerDied","Data":"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400"} Oct 30 21:58:32.834599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.828655 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" event={"ID":"3be7072c-9090-4137-a303-45d5f2e3191d","Type":"ContainerDied","Data":"ffd85d77f5673837faf9d3fb4d14d5e0193c79452c5f4eccfaabad63e22ea0b2"} Oct 30 21:58:32.834599 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.832735 4525 scope.go:122] "RemoveContainer" containerID="fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400" Oct 30 21:58:32.908816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.908787 4525 scope.go:122] "RemoveContainer" containerID="fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400" Oct 30 21:58:32.910901 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:32.910867 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400\": container with ID starting with fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400 not found: ID does not exist" containerID="fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400" Oct 30 21:58:32.911059 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.910914 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400"} err="failed to get container status \"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400\": rpc error: code = NotFound desc = could not find container \"fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400\": container with ID starting with fbb92f7a43b0236bd1a5eab91eab9c0f044d696e6c8bfee593ec0856d49c3400 not found: ID does not exist" Oct 30 21:58:32.969232 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.969195 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\") pod \"3be7072c-9090-4137-a303-45d5f2e3191d\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " Oct 30 21:58:32.969422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.969271 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") pod \"3be7072c-9090-4137-a303-45d5f2e3191d\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " Oct 30 21:58:32.969422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.969364 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\") pod \"3be7072c-9090-4137-a303-45d5f2e3191d\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " Oct 30 21:58:32.969422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.969399 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\" (UniqueName: \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\") pod \"3be7072c-9090-4137-a303-45d5f2e3191d\" (UID: \"3be7072c-9090-4137-a303-45d5f2e3191d\") " Oct 30 21:58:32.971372 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.970219 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca" pod "3be7072c-9090-4137-a303-45d5f2e3191d" (UID: "3be7072c-9090-4137-a303-45d5f2e3191d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:32.973900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.973861 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2" pod "3be7072c-9090-4137-a303-45d5f2e3191d" (UID: "3be7072c-9090-4137-a303-45d5f2e3191d"). InnerVolumeSpecName "kube-api-access-t2hq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:32.974139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.974114 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs" pod "3be7072c-9090-4137-a303-45d5f2e3191d" (UID: "3be7072c-9090-4137-a303-45d5f2e3191d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:32.976103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:32.976069 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls" pod "3be7072c-9090-4137-a303-45d5f2e3191d" (UID: "3be7072c-9090-4137-a303-45d5f2e3191d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:33.071529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.071447 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.071529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.071478 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be7072c-9090-4137-a303-45d5f2e3191d-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.071529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.071490 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hq2\" (UniqueName: \"kubernetes.io/projected/3be7072c-9090-4137-a303-45d5f2e3191d-kube-api-access-t2hq2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.071529 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.071500 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be7072c-9090-4137-a303-45d5f2e3191d-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:33.836988 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.836904 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj" Oct 30 21:58:33.965342 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.965307 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:58:33.965521 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:33.965489 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" podUID="86a87117-4564-47a5-a067-8d072b119395" containerName="maas-api" containerID="cri-o://80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8" gracePeriod=30 Oct 30 21:58:34.003297 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.003267 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:58:34.016106 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.016072 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d6e0be-54dd49fbdb-86qnj"] Oct 30 21:58:34.297633 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.294723 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:58:34.391724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.391317 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") pod \"86a87117-4564-47a5-a067-8d072b119395\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " Oct 30 21:58:34.391724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.391421 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\") pod \"86a87117-4564-47a5-a067-8d072b119395\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " Oct 30 21:58:34.391724 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.391550 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\") pod \"86a87117-4564-47a5-a067-8d072b119395\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " Oct 30 21:58:34.392068 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.391741 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\") pod \"86a87117-4564-47a5-a067-8d072b119395\" (UID: \"86a87117-4564-47a5-a067-8d072b119395\") " Oct 30 21:58:34.449841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.443952 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl" pod "86a87117-4564-47a5-a067-8d072b119395" (UID: "86a87117-4564-47a5-a067-8d072b119395"). InnerVolumeSpecName "kube-api-access-58xkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:34.449841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.444400 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls" pod "86a87117-4564-47a5-a067-8d072b119395" (UID: "86a87117-4564-47a5-a067-8d072b119395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:34.462120 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.459795 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca" pod "86a87117-4564-47a5-a067-8d072b119395" (UID: "86a87117-4564-47a5-a067-8d072b119395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:34.475914 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.473445 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs" pod "86a87117-4564-47a5-a067-8d072b119395" (UID: "86a87117-4564-47a5-a067-8d072b119395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:34.494957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.494737 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a87117-4564-47a5-a067-8d072b119395-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:34.494957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.494775 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:34.494957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.494789 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xkl\" (UniqueName: \"kubernetes.io/projected/86a87117-4564-47a5-a067-8d072b119395-kube-api-access-58xkl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:34.494957 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.494806 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a87117-4564-47a5-a067-8d072b119395-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:34.845501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.845420 4525 generic.go:356] "Generic (PLEG): container finished" podID="86a87117-4564-47a5-a067-8d072b119395" containerID="80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8" exitCode=0 Oct 30 21:58:34.845501 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.845466 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" Oct 30 21:58:34.846029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.845494 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" event={"ID":"86a87117-4564-47a5-a067-8d072b119395","Type":"ContainerDied","Data":"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8"} Oct 30 21:58:34.846029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.845551 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr" event={"ID":"86a87117-4564-47a5-a067-8d072b119395","Type":"ContainerDied","Data":"3fcdc80937eafe18f7a1ec58c394bd18919d47164d7fd14a49843dd773e5eb6c"} Oct 30 21:58:34.846029 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.845577 4525 scope.go:122] "RemoveContainer" containerID="80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8" Oct 30 21:58:34.856333 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.856312 4525 scope.go:122] "RemoveContainer" containerID="80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8" Oct 30 21:58:34.856643 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:34.856620 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8\": container with ID starting with 80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8 not found: ID does not exist" containerID="80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8" Oct 30 21:58:34.856722 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:34.856656 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8"} err="failed to get container status \"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8\": rpc error: code = NotFound desc = could not find container \"80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8\": container with ID starting with 80e737baf11bf3f6fcb0d658a64a9eaee7d3208972a4f4c16c95e022d30fc8c8 not found: ID does not exist" Oct 30 21:58:35.029784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:35.025651 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:58:35.033627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:35.030547 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e51c52-bc8784ddd-kntwr"] Oct 30 21:58:35.397002 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:35.396964 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:58:35.474304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:35.474265 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be7072c-9090-4137-a303-45d5f2e3191d" path="/var/lib/kubelet/pods/3be7072c-9090-4137-a303-45d5f2e3191d/volumes" Oct 30 21:58:35.474761 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:35.474745 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a87117-4564-47a5-a067-8d072b119395" path="/var/lib/kubelet/pods/86a87117-4564-47a5-a067-8d072b119395/volumes" Oct 30 21:58:35.845396 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:35.845300 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" podUID="e54b0950-0cef-44bb-b04b-677f0acbb6db" containerName="payload-pre-processing" containerID="cri-o://fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f" gracePeriod=25 Oct 30 21:58:36.248021 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.247994 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:58:36.343243 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.343202 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\" (UniqueName: \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\") pod \"e54b0950-0cef-44bb-b04b-677f0acbb6db\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " Oct 30 21:58:36.343424 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.343285 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\") pod \"e54b0950-0cef-44bb-b04b-677f0acbb6db\" (UID: \"e54b0950-0cef-44bb-b04b-677f0acbb6db\") " Oct 30 21:58:36.343726 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.343701 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume" pod "e54b0950-0cef-44bb-b04b-677f0acbb6db" (UID: "e54b0950-0cef-44bb-b04b-677f0acbb6db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:36.345704 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.345681 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5" pod "e54b0950-0cef-44bb-b04b-677f0acbb6db" (UID: "e54b0950-0cef-44bb-b04b-677f0acbb6db"). InnerVolumeSpecName "kube-api-access-m5nn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:36.445181 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.445141 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54b0950-0cef-44bb-b04b-677f0acbb6db-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:36.445181 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.445174 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5nn5\" (UniqueName: \"kubernetes.io/projected/e54b0950-0cef-44bb-b04b-677f0acbb6db-kube-api-access-m5nn5\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:36.868439 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.868396 4525 generic.go:356] "Generic (PLEG): container finished" podID="e54b0950-0cef-44bb-b04b-677f0acbb6db" containerID="fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f" exitCode=0 Oct 30 21:58:36.868669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.868442 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" Oct 30 21:58:36.868669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.868482 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" event={"ID":"e54b0950-0cef-44bb-b04b-677f0acbb6db","Type":"ContainerDied","Data":"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f"} Oct 30 21:58:36.868669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.868544 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls" event={"ID":"e54b0950-0cef-44bb-b04b-677f0acbb6db","Type":"ContainerDied","Data":"af89d0a9f9f02d8309c42add3c2bfa9390e82d1f58ce74cbfef2e73098ba02a8"} Oct 30 21:58:36.868669 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.868569 4525 scope.go:122] "RemoveContainer" containerID="fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f" Oct 30 21:58:36.878723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.878705 4525 scope.go:122] "RemoveContainer" containerID="fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f" Oct 30 21:58:36.878998 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:36.878977 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f\": container with ID starting with fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f not found: ID does not exist" containerID="fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f" Oct 30 21:58:36.879049 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:36.879007 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f"} err="failed to get container status \"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f\": rpc error: code = NotFound desc = could not find container \"fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f\": container with ID starting with fe6fae81b38b6fff50e3ff13f352b131c15345c7bb14f1e5f232b18f0b2aaa4f not found: ID does not exist" Oct 30 21:58:37.039449 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:37.039411 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:58:37.089116 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:37.089074 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bd3c59-6777796f9-7ffls"] Oct 30 21:58:37.476994 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:37.476960 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54b0950-0cef-44bb-b04b-677f0acbb6db" path="/var/lib/kubelet/pods/e54b0950-0cef-44bb-b04b-677f0acbb6db/volumes" Oct 30 21:58:40.638524 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:40.638494 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:58:40.638976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:40.638713 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" podUID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" containerName="maas-api" containerID="cri-o://770cf69ae21e8a95b34eb59c5a0c4250511fccbac1b75dcf88576a63203ea095" gracePeriod=30 Oct 30 21:58:40.935587 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:40.935496 4525 generic.go:356] "Generic (PLEG): container finished" podID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" containerID="770cf69ae21e8a95b34eb59c5a0c4250511fccbac1b75dcf88576a63203ea095" exitCode=0 Oct 30 21:58:40.935806 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:40.935732 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" event={"ID":"05cffc26-74ce-4e79-93ad-8ba262b09e6f","Type":"ContainerDied","Data":"770cf69ae21e8a95b34eb59c5a0c4250511fccbac1b75dcf88576a63203ea095"} Oct 30 21:58:40.975560 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:40.975530 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:58:41.144090 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.144049 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") pod \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " Oct 30 21:58:41.145060 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.145032 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\") pod \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " Oct 30 21:58:41.145925 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.145903 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\" (UniqueName: \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\") pod \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " Oct 30 21:58:41.147317 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.147258 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca" pod "05cffc26-74ce-4e79-93ad-8ba262b09e6f" (UID: "05cffc26-74ce-4e79-93ad-8ba262b09e6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:41.147507 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.147488 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\") pod \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\" (UID: \"05cffc26-74ce-4e79-93ad-8ba262b09e6f\") " Oct 30 21:58:41.152820 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.152795 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cffc26-74ce-4e79-93ad-8ba262b09e6f-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.163662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.162034 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls" pod "05cffc26-74ce-4e79-93ad-8ba262b09e6f" (UID: "05cffc26-74ce-4e79-93ad-8ba262b09e6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:41.163662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.163585 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs" pod "05cffc26-74ce-4e79-93ad-8ba262b09e6f" (UID: "05cffc26-74ce-4e79-93ad-8ba262b09e6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:41.163906 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.163881 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr" pod "05cffc26-74ce-4e79-93ad-8ba262b09e6f" (UID: "05cffc26-74ce-4e79-93ad-8ba262b09e6f"). InnerVolumeSpecName "kube-api-access-d4fpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:41.254668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.254572 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.254668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.254599 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05cffc26-74ce-4e79-93ad-8ba262b09e6f-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.254668 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.254651 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4fpr\" (UniqueName: \"kubernetes.io/projected/05cffc26-74ce-4e79-93ad-8ba262b09e6f-kube-api-access-d4fpr\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.944304 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.944273 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" Oct 30 21:58:41.944784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.944318 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v" event={"ID":"05cffc26-74ce-4e79-93ad-8ba262b09e6f","Type":"ContainerDied","Data":"84105c47cd0e5f0593ddf6ea171b4d7b004143561f64cc0421e18592b6abdf3b"} Oct 30 21:58:41.944784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:41.944395 4525 scope.go:122] "RemoveContainer" containerID="770cf69ae21e8a95b34eb59c5a0c4250511fccbac1b75dcf88576a63203ea095" Oct 30 21:58:42.112976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:42.112944 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:58:42.163051 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:42.163019 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c92783-5555445c67-9jn8v"] Oct 30 21:58:43.216528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:43.216479 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:58:43.219000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:43.216769 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" containerID="cri-o://f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315" gracePeriod=30 Oct 30 21:58:43.474118 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:43.474038 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" path="/var/lib/kubelet/pods/05cffc26-74ce-4e79-93ad-8ba262b09e6f/volumes" Oct 30 21:58:48.550181 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.550153 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:58:48.597905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.597868 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598077 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.597998 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598144 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598080 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598144 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598119 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598152 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598172 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598214 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598244 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598197 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:48.598505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598258 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:48.598505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598289 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598325 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\") pod \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\" (UID: \"00bf9cc7-9c21-482d-be72-89afa7e05dd3\") " Oct 30 21:58:48.598505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598377 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:48.598505 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598454 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:48.598811 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.598628 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:48.599326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.599211 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.599326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.599242 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.599326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.599258 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.599326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.599273 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.599326 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.599288 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.600960 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.600845 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:48.601078 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.601045 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:48.601168 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.601150 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "kube-api-access-n5m4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:48.601233 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.601176 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy" pod "00bf9cc7-9c21-482d-be72-89afa7e05dd3" (UID: "00bf9cc7-9c21-482d-be72-89afa7e05dd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:48.723639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.706884 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.723639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.706925 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.723639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.706939 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5m4z\" (UniqueName: \"kubernetes.io/projected/00bf9cc7-9c21-482d-be72-89afa7e05dd3-kube-api-access-n5m4z\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:48.723639 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:48.706953 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bf9cc7-9c21-482d-be72-89afa7e05dd3-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:49.057809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.057772 4525 generic.go:356] "Generic (PLEG): container finished" podID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerID="f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315" exitCode=0 Oct 30 21:58:49.058012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.057837 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" Oct 30 21:58:49.058012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.057849 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerDied","Data":"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315"} Oct 30 21:58:49.058012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.057910 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj" event={"ID":"00bf9cc7-9c21-482d-be72-89afa7e05dd3","Type":"ContainerDied","Data":"775eab5e8721ef3ce246e87bf109b6d3631881e989afbaffd597499d642176f7"} Oct 30 21:58:49.058012 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.057933 4525 scope.go:122] "RemoveContainer" containerID="f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315" Oct 30 21:58:49.059352 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.059243 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:58:49.075404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.075385 4525 scope.go:122] "RemoveContainer" containerID="cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256" Oct 30 21:58:49.085627 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.085595 4525 scope.go:122] "RemoveContainer" containerID="f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315" Oct 30 21:58:49.085946 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:49.085923 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315\": container with ID starting with f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315 not found: ID does not exist" containerID="f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315" Oct 30 21:58:49.086008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.085959 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315"} err="failed to get container status \"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315\": rpc error: code = NotFound desc = could not find container \"f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315\": container with ID starting with f7ee3796ae2fa17d5a20c13228d678e656389feb668b2d5425c620739424a315 not found: ID does not exist" Oct 30 21:58:49.086008 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.085979 4525 scope.go:122] "RemoveContainer" containerID="cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256" Oct 30 21:58:49.086261 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:49.086240 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256\": container with ID starting with cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256 not found: ID does not exist" containerID="cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256" Oct 30 21:58:49.086319 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.086268 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256"} err="failed to get container status \"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256\": rpc error: code = NotFound desc = could not find container \"cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256\": container with ID starting with cfa6c9cdd67a4ade9fdc27c7788b85a8b6583222616281ffec2ffb7dd2f35256 not found: ID does not exist" Oct 30 21:58:49.235473 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.235437 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:58:49.255247 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.255206 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d6e0be-openshift-default-f886c67f-qb7tj"] Oct 30 21:58:49.474222 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:49.474192 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" path="/var/lib/kubelet/pods/00bf9cc7-9c21-482d-be72-89afa7e05dd3/volumes" Oct 30 21:58:50.526621 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:50.526575 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:58:50.527113 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:50.526839 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" containerID="cri-o://ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5" gracePeriod=30 Oct 30 21:58:55.786069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.786031 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:58:55.818696 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.818664 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:58:55.895672 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895641 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.895865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895696 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.895865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895736 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.895865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895766 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.895865 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895810 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.896071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895883 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.896071 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.895966 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:55.896189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896071 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.896189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896066 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:55.896189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896111 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.896189 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896156 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\") pod \"fc858730-9758-465a-b392-ea601fcc791d\" (UID: \"fc858730-9758-465a-b392-ea601fcc791d\") " Oct 30 21:58:55.896404 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896356 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:55.896464 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896406 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:55.896597 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896543 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:55.896813 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896798 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.896878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896815 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.896878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896825 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.896878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896834 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc858730-9758-465a-b392-ea601fcc791d-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.896878 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.896842 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.898230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.898200 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "kube-api-access-7927h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:55.898230 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.898212 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:55.898387 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.898291 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:55.898583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.898564 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo" pod "fc858730-9758-465a-b392-ea601fcc791d" (UID: "fc858730-9758-465a-b392-ea601fcc791d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:55.998141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.998099 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc858730-9758-465a-b392-ea601fcc791d-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.998141 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.998137 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.998342 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.998151 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7927h\" (UniqueName: \"kubernetes.io/projected/fc858730-9758-465a-b392-ea601fcc791d-kube-api-access-7927h\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:55.998342 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:55.998165 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc858730-9758-465a-b392-ea601fcc791d-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:56.152636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.152459 4525 generic.go:356] "Generic (PLEG): container finished" podID="fc858730-9758-465a-b392-ea601fcc791d" containerID="ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5" exitCode=0 Oct 30 21:58:56.152636 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.152600 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerDied","Data":"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5"} Oct 30 21:58:56.152851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.152711 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" event={"ID":"fc858730-9758-465a-b392-ea601fcc791d","Type":"ContainerDied","Data":"1faeaa3fa5022552fb94a0e23a07598c9d19292e4408386f0f53c42615e32833"} Oct 30 21:58:56.152851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.152743 4525 scope.go:122] "RemoveContainer" containerID="ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5" Oct 30 21:58:56.152851 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.152771 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht" Oct 30 21:58:56.167049 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.167024 4525 scope.go:122] "RemoveContainer" containerID="0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb" Oct 30 21:58:56.177323 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.177303 4525 scope.go:122] "RemoveContainer" containerID="ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5" Oct 30 21:58:56.177649 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:56.177600 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5\": container with ID starting with ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5 not found: ID does not exist" containerID="ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5" Oct 30 21:58:56.177829 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.177798 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5"} err="failed to get container status \"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5\": rpc error: code = NotFound desc = could not find container \"ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5\": container with ID starting with ff51b77421b27be242837e47d7126f4f34c50ac3c071b5eade830a4e61b517a5 not found: ID does not exist" Oct 30 21:58:56.177927 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.177832 4525 scope.go:122] "RemoveContainer" containerID="0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb" Oct 30 21:58:56.178146 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:56.178126 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb\": container with ID starting with 0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb not found: ID does not exist" containerID="0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb" Oct 30 21:58:56.178208 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.178151 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb"} err="failed to get container status \"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb\": rpc error: code = NotFound desc = could not find container \"0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb\": container with ID starting with 0eae9a2509f912734bf7c3882639f9ddcd7532cd444c45e9c156f16007f756cb not found: ID does not exist" Oct 30 21:58:56.338476 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.338439 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:58:56.342133 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:56.342103 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c92783-openshift-default-7d6987cf49-s8sht"] Oct 30 21:58:57.449148 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.449110 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:58:57.453035 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.452454 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" podUID="069c580a-1bb3-4bf6-a344-0c54b3145a30" containerName="maas-api" containerID="cri-o://89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a" gracePeriod=30 Oct 30 21:58:57.504978 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.504942 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc858730-9758-465a-b392-ea601fcc791d" path="/var/lib/kubelet/pods/fc858730-9758-465a-b392-ea601fcc791d/volumes" Oct 30 21:58:57.795039 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.795011 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:58:57.947616 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.947579 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\" (UniqueName: \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\") pod \"069c580a-1bb3-4bf6-a344-0c54b3145a30\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " Oct 30 21:58:57.947799 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.947733 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\") pod \"069c580a-1bb3-4bf6-a344-0c54b3145a30\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " Oct 30 21:58:57.948004 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.947983 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\") pod \"069c580a-1bb3-4bf6-a344-0c54b3145a30\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " Oct 30 21:58:57.948104 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.948034 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") pod \"069c580a-1bb3-4bf6-a344-0c54b3145a30\" (UID: \"069c580a-1bb3-4bf6-a344-0c54b3145a30\") " Oct 30 21:58:57.948415 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.948337 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca" pod "069c580a-1bb3-4bf6-a344-0c54b3145a30" (UID: "069c580a-1bb3-4bf6-a344-0c54b3145a30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:57.948974 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.948941 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069c580a-1bb3-4bf6-a344-0c54b3145a30-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:57.950528 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.950484 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs" pod "069c580a-1bb3-4bf6-a344-0c54b3145a30" (UID: "069c580a-1bb3-4bf6-a344-0c54b3145a30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:57.950681 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.950653 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl" pod "069c580a-1bb3-4bf6-a344-0c54b3145a30" (UID: "069c580a-1bb3-4bf6-a344-0c54b3145a30"). InnerVolumeSpecName "kube-api-access-wg9dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:57.950905 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:57.950847 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls" pod "069c580a-1bb3-4bf6-a344-0c54b3145a30" (UID: "069c580a-1bb3-4bf6-a344-0c54b3145a30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:58.055190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.054770 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.055190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.054807 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9dl\" (UniqueName: \"kubernetes.io/projected/069c580a-1bb3-4bf6-a344-0c54b3145a30-kube-api-access-wg9dl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.055190 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.054824 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069c580a-1bb3-4bf6-a344-0c54b3145a30-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:58.182294 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.182254 4525 generic.go:356] "Generic (PLEG): container finished" podID="069c580a-1bb3-4bf6-a344-0c54b3145a30" containerID="89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a" exitCode=0 Oct 30 21:58:58.182471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.182290 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" event={"ID":"069c580a-1bb3-4bf6-a344-0c54b3145a30","Type":"ContainerDied","Data":"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a"} Oct 30 21:58:58.182471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.182321 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" Oct 30 21:58:58.182471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.182351 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9" event={"ID":"069c580a-1bb3-4bf6-a344-0c54b3145a30","Type":"ContainerDied","Data":"a89fbaff315051f29d5c0395f36b1138cd82b1ef0828afaa1325b146555634d7"} Oct 30 21:58:58.182471 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.182378 4525 scope.go:122] "RemoveContainer" containerID="89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a" Oct 30 21:58:58.196422 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.196381 4525 scope.go:122] "RemoveContainer" containerID="89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a" Oct 30 21:58:58.196735 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:58:58.196711 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a\": container with ID starting with 89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a not found: ID does not exist" containerID="89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a" Oct 30 21:58:58.196816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.196747 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a"} err="failed to get container status \"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a\": rpc error: code = NotFound desc = could not find container \"89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a\": container with ID starting with 89db4efe2e594b3bba416375f837d131352ac7149a2f08e2ec478cd1bf22c35a not found: ID does not exist" Oct 30 21:58:58.348590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.348497 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:58:58.352506 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:58.352474 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c632-c9c6bc9f8-qxgl9"] Oct 30 21:58:59.481561 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:58:59.481516 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069c580a-1bb3-4bf6-a344-0c54b3145a30" path="/var/lib/kubelet/pods/069c580a-1bb3-4bf6-a344-0c54b3145a30/volumes" Oct 30 21:59:00.787338 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:00.787299 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" podUID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" containerName="payload-pre-processing" containerID="cri-o://2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493" gracePeriod=25 Oct 30 21:59:01.074956 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.074915 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:59:01.198095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.198060 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\" (UniqueName: \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\") pod \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " Oct 30 21:59:01.198310 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.198291 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\") pod \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\" (UID: \"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21\") " Oct 30 21:59:01.199148 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.199116 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume" pod "b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" (UID: "b8ca5322-49c1-4f2a-84f0-c2408ccfaa21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:01.201197 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.201165 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76" pod "b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" (UID: "b8ca5322-49c1-4f2a-84f0-c2408ccfaa21"). InnerVolumeSpecName "kube-api-access-5qg76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:01.218399 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.218367 4525 generic.go:356] "Generic (PLEG): container finished" podID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" containerID="2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493" exitCode=0 Oct 30 21:59:01.218583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.218475 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" Oct 30 21:59:01.218583 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.218513 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" event={"ID":"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21","Type":"ContainerDied","Data":"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493"} Oct 30 21:59:01.218784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.218587 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w" event={"ID":"b8ca5322-49c1-4f2a-84f0-c2408ccfaa21","Type":"ContainerDied","Data":"9cbccd1949aacca4d4c3afa8141143f3fa83a4b18440d7a8612c6e5d01801a85"} Oct 30 21:59:01.218784 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.218650 4525 scope.go:122] "RemoveContainer" containerID="2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493" Oct 30 21:59:01.231816 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.231790 4525 scope.go:122] "RemoveContainer" containerID="2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493" Oct 30 21:59:01.232122 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:01.232101 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493\": container with ID starting with 2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493 not found: ID does not exist" containerID="2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493" Oct 30 21:59:01.232216 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.232130 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493"} err="failed to get container status \"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493\": rpc error: code = NotFound desc = could not find container \"2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493\": container with ID starting with 2e5fe48bde73cf047cd5fe4635088e59c0aa9ace52494aa3e5cd6611f9e1c493 not found: ID does not exist" Oct 30 21:59:01.300143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.300066 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.300143 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.300095 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg76\" (UniqueName: \"kubernetes.io/projected/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21-kube-api-access-5qg76\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.384991 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.384958 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:59:01.388665 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.388639 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78c632-86565ffd99-7n62w"] Oct 30 21:59:01.474139 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:01.474100 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" path="/var/lib/kubelet/pods/b8ca5322-49c1-4f2a-84f0-c2408ccfaa21/volumes" Oct 30 21:59:05.319762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.319723 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 21:59:05.320373 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320356 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320377 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320388 4525 cpu_manager.go:414] "containerMap: removing container" podUID="86a87117-4564-47a5-a067-8d072b119395" containerName="maas-api" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320393 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a87117-4564-47a5-a067-8d072b119395" containerName="maas-api" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320402 4525 cpu_manager.go:414] "containerMap: removing container" podUID="3be7072c-9090-4137-a303-45d5f2e3191d" containerName="maas-api" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320408 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be7072c-9090-4137-a303-45d5f2e3191d" containerName="maas-api" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320414 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" containerName="payload-pre-processing" Oct 30 21:59:05.320420 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320419 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320428 4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320433 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320451 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320458 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320465 4525 cpu_manager.go:414] "containerMap: removing container" podUID="069c580a-1bb3-4bf6-a344-0c54b3145a30" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320470 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069c580a-1bb3-4bf6-a344-0c54b3145a30" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320481 4525 cpu_manager.go:414] "containerMap: removing container" podUID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320493 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320505 4525 cpu_manager.go:414] "containerMap: removing container" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320509 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320519 4525 cpu_manager.go:414] "containerMap: removing container" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320523 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320531 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320536 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" containerName="maas-api" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320559 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e54b0950-0cef-44bb-b04b-677f0acbb6db" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320564 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54b0950-0cef-44bb-b04b-677f0acbb6db" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320663 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ca5322-49c1-4f2a-84f0-c2408ccfaa21" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320674 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf48bde5-b0d7-4c9c-ac0f-38eb63380cdd" containerName="payload-pre-processing" Oct 30 21:59:05.320679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320683 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320692 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="069c580a-1bb3-4bf6-a344-0c54b3145a30" containerName="maas-api" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320703 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cffc26-74ce-4e79-93ad-8ba262b09e6f" containerName="maas-api" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320710 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320716 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bf9cc7-9c21-482d-be72-89afa7e05dd3" containerName="istio-proxy" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320721 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b931a0-fa52-4f2a-a309-b21c939a70d7" containerName="maas-api" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320727 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54b0950-0cef-44bb-b04b-677f0acbb6db" containerName="payload-pre-processing" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320735 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc858730-9758-465a-b392-ea601fcc791d" containerName="istio-proxy" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320740 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a87117-4564-47a5-a067-8d072b119395" containerName="maas-api" Oct 30 21:59:05.321276 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.320744 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be7072c-9090-4137-a303-45d5f2e3191d" containerName="maas-api" Oct 30 21:59:05.325646 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.325600 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.381984 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.381951 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 21:59:05.402916 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:05.402875 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:59:05.460632 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.447744 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8c5350\"" Oct 30 21:59:05.506628 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.473400 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8c5350-dockercfg-zcwtm\"" Oct 30 21:59:05.506892 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.496225 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 21:59:05.520413 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.520378 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.524530 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.524495 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.524921 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.524895 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zpd\" (UniqueName: \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.594644 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.594492 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 21:59:05.626472 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.626431 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxlw\" (UniqueName: \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.626723 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.626664 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.626809 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.626736 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.626891 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.626858 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zpd\" (UniqueName: \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.628145 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.628114 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.688710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.688677 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zpd\" (UniqueName: \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\") pod \"payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:05.727830 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.727793 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.728055 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.728030 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxlw\" (UniqueName: \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.728558 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.728539 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.789730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.789693 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxlw\" (UniqueName: \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\") pod \"payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.913000 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.912962 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:05.940048 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:05.940017 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:06.292746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:06.292718 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 21:59:06.292959 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:59:06.292937 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d16570_6f18_4bba_862f_0303560cfe33.slice/crio-991135cdeb78d1559e1bc8a6645fef294260d6ad38ff97400bdefef3214d3526 WatchSource:0}: Error finding container 991135cdeb78d1559e1bc8a6645fef294260d6ad38ff97400bdefef3214d3526: Status 404 returned error can't find the container with id 991135cdeb78d1559e1bc8a6645fef294260d6ad38ff97400bdefef3214d3526 Oct 30 21:59:06.344623 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:06.344577 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 21:59:07.302337 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.302298 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" event={"ID":"1af2106e-af7b-4c0b-a81a-ffd125a99e06","Type":"ContainerStarted","Data":"3ae9fa17c5d842118b1015d544fc6cb76fefaab5ff450fb82a52a4467e24f1e7"} Oct 30 21:59:07.302542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.302382 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" event={"ID":"1af2106e-af7b-4c0b-a81a-ffd125a99e06","Type":"ContainerStarted","Data":"980d1bcec8a1d07b6b1864d77bc70673982a0cb5ac5779803ff0d07b577aa705"} Oct 30 21:59:07.302542 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.302406 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:07.304274 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.304247 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" event={"ID":"92d16570-6f18-4bba-862f-0303560cfe33","Type":"ContainerStarted","Data":"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2"} Oct 30 21:59:07.304421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.304305 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:07.304421 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.304327 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" event={"ID":"92d16570-6f18-4bba-862f-0303560cfe33","Type":"ContainerStarted","Data":"991135cdeb78d1559e1bc8a6645fef294260d6ad38ff97400bdefef3214d3526"} Oct 30 21:59:07.422003 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.421950 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" podStartSLOduration=2.4219349980000002 podStartE2EDuration="2.421934998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:07.417980536 +0000 UTC m=+2172.092027073" watchObservedRunningTime="2026-10-30 21:59:07.421934998 +0000 UTC m=+2172.095981533" Oct 30 21:59:07.567679 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:07.567503 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" podStartSLOduration=2.567479858 podStartE2EDuration="2.567479858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:07.560237919 +0000 UTC m=+2172.234284456" watchObservedRunningTime="2026-10-30 21:59:07.567479858 +0000 UTC m=+2172.241526395" Oct 30 21:59:09.787041 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.787008 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 21:59:09.791487 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.791467 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.895768 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.895734 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fd2a94-serving-cert\"" Oct 30 21:59:09.896797 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.896772 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.896896 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.896810 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.896976 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.896950 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbh6\" (UniqueName: \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.897103 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.897084 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.899298 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.899277 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 21:59:09.998235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.998197 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.998235 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.998243 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.998511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.998302 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbh6\" (UniqueName: \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.998511 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.998371 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:09.998511 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:09.998499 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-serving-cert: secret "maas-api-e2e-inf-a-fd2a94-serving-cert" not found Oct 30 21:59:09.998709 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:09.998577 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls podName:abe516fb-955b-4a93-8dd8-63c2bd851b8c nodeName:}" failed. No retries permitted until 2026-10-30 21:59:10.498559855 +0000 UTC m=+2175.172606373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls") pod "maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" (UID: "abe516fb-955b-4a93-8dd8-63c2bd851b8c") : secret "maas-api-e2e-inf-a-fd2a94-serving-cert" not found Oct 30 21:59:09.999070 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:09.999051 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:10.000962 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:10.000937 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:10.060795 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:10.060720 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbh6\" (UniqueName: \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:10.507369 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:10.507323 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:10.509894 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:10.509865 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:10.706575 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:10.706547 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:11.137972 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:11.137943 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 21:59:11.139255 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:59:11.139225 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe516fb_955b_4a93_8dd8_63c2bd851b8c.slice/crio-f0b5db6ac91e2a827f1a05eedf78efb67381f64ec7621b39f9a11c93e0e3b594 WatchSource:0}: Error finding container f0b5db6ac91e2a827f1a05eedf78efb67381f64ec7621b39f9a11c93e0e3b594: Status 404 returned error can't find the container with id f0b5db6ac91e2a827f1a05eedf78efb67381f64ec7621b39f9a11c93e0e3b594 Oct 30 21:59:11.140902 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:11.140881 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:59:11.330460 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:11.330426 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" event={"ID":"abe516fb-955b-4a93-8dd8-63c2bd851b8c","Type":"ContainerStarted","Data":"f0b5db6ac91e2a827f1a05eedf78efb67381f64ec7621b39f9a11c93e0e3b594"} Oct 30 21:59:12.338325 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:12.338236 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" event={"ID":"abe516fb-955b-4a93-8dd8-63c2bd851b8c","Type":"ContainerStarted","Data":"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43"} Oct 30 21:59:12.338812 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:12.338374 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:12.456820 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:12.456756 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" podStartSLOduration=2.585417481 podStartE2EDuration="3.456736911s" totalImagesPullingTime="871.31943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:12.456063168 +0000 UTC m=+2177.130109704" watchObservedRunningTime="2026-10-30 21:59:12.456736911 +0000 UTC m=+2177.130783447" Oct 30 21:59:13.250719 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:13.250681 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:59:13.366955 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:13.366908 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:59:18.251732 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.251663 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" podUID="4797e46e-7da8-46f8-8b8f-77109e06e63f" containerName="payload-processing" containerID="cri-o://5963c72a6f4075dcb384bc586bc5503d32e1c8184f7507edb8533df443e2ab6e" gracePeriod=25 Oct 30 21:59:18.314462 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.314436 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 21:59:18.314643 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.314500 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 21:59:18.351098 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.351074 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 21:59:18.367802 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.367770 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" podUID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" containerName="payload-pre-processing" containerID="cri-o://8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df" gracePeriod=25 Oct 30 21:59:18.379438 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.379409 4525 generic.go:356] "Generic (PLEG): container finished" podID="4797e46e-7da8-46f8-8b8f-77109e06e63f" containerID="5963c72a6f4075dcb384bc586bc5503d32e1c8184f7507edb8533df443e2ab6e" exitCode=0 Oct 30 21:59:18.379590 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.379488 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" event={"ID":"4797e46e-7da8-46f8-8b8f-77109e06e63f","Type":"ContainerDied","Data":"5963c72a6f4075dcb384bc586bc5503d32e1c8184f7507edb8533df443e2ab6e"} Oct 30 21:59:18.562279 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.562254 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:59:18.674400 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.674370 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\" (UniqueName: \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\") pod \"4797e46e-7da8-46f8-8b8f-77109e06e63f\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " Oct 30 21:59:18.674552 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.674427 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\") pod \"4797e46e-7da8-46f8-8b8f-77109e06e63f\" (UID: \"4797e46e-7da8-46f8-8b8f-77109e06e63f\") " Oct 30 21:59:18.674841 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.674814 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume" pod "4797e46e-7da8-46f8-8b8f-77109e06e63f" (UID: "4797e46e-7da8-46f8-8b8f-77109e06e63f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.674989 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.674973 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4797e46e-7da8-46f8-8b8f-77109e06e63f-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.676674 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.676649 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll" pod "4797e46e-7da8-46f8-8b8f-77109e06e63f" (UID: "4797e46e-7da8-46f8-8b8f-77109e06e63f"). InnerVolumeSpecName "kube-api-access-8wmll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.705069 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.689940 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:59:18.776101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.776017 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\") pod \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " Oct 30 21:59:18.776257 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.776186 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\" (UniqueName: \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\") pod \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\" (UID: \"a52b3140-9db4-4f51-84d4-ce72106b5bc7\") " Oct 30 21:59:18.776392 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.776370 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume" pod "a52b3140-9db4-4f51-84d4-ce72106b5bc7" (UID: "a52b3140-9db4-4f51-84d4-ce72106b5bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.776662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.776640 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wmll\" (UniqueName: \"kubernetes.io/projected/4797e46e-7da8-46f8-8b8f-77109e06e63f-kube-api-access-8wmll\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.776662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.776658 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a52b3140-9db4-4f51-84d4-ce72106b5bc7-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.778509 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.778489 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k" pod "a52b3140-9db4-4f51-84d4-ce72106b5bc7" (UID: "a52b3140-9db4-4f51-84d4-ce72106b5bc7"). InnerVolumeSpecName "kube-api-access-vjl4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.878264 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:18.878223 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjl4k\" (UniqueName: \"kubernetes.io/projected/a52b3140-9db4-4f51-84d4-ce72106b5bc7-kube-api-access-vjl4k\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:19.386706 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.386677 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" Oct 30 21:59:19.387215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.386702 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp" event={"ID":"4797e46e-7da8-46f8-8b8f-77109e06e63f","Type":"ContainerDied","Data":"e76b707a0047c2e7b4a766d818de021136f4ba1c702c9a16ed0f5b4b6e8d24f8"} Oct 30 21:59:19.387215 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.386794 4525 scope.go:122] "RemoveContainer" containerID="5963c72a6f4075dcb384bc586bc5503d32e1c8184f7507edb8533df443e2ab6e" Oct 30 21:59:19.388480 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.388457 4525 generic.go:356] "Generic (PLEG): container finished" podID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" containerID="8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df" exitCode=0 Oct 30 21:59:19.388657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.388523 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" event={"ID":"a52b3140-9db4-4f51-84d4-ce72106b5bc7","Type":"ContainerDied","Data":"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df"} Oct 30 21:59:19.388657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.388560 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" event={"ID":"a52b3140-9db4-4f51-84d4-ce72106b5bc7","Type":"ContainerDied","Data":"5ffbbf62c0e18e5636daa70a8c64d32d97c25b2ae848575dcb517af1fe9b2fc1"} Oct 30 21:59:19.388657 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.388588 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj" Oct 30 21:59:19.398052 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.398032 4525 scope.go:122] "RemoveContainer" containerID="8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df" Oct 30 21:59:19.407932 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.407913 4525 scope.go:122] "RemoveContainer" containerID="8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df" Oct 30 21:59:19.408267 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:19.408246 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df\": container with ID starting with 8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df not found: ID does not exist" containerID="8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df" Oct 30 21:59:19.408315 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.408275 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df"} err="failed to get container status \"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df\": rpc error: code = NotFound desc = could not find container \"8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df\": container with ID starting with 8a04c8f6abecc89368fb8baf13554e186c7d8e9de051b3e6c6dbd0fbddb988df not found: ID does not exist" Oct 30 21:59:19.602430 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.602338 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:59:19.608985 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.608959 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5bfa09-66bbf6887d-snvgp"] Oct 30 21:59:19.717128 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.717090 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:59:19.720364 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:19.720339 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5bfa09-76895fd6dc-l5hdj"] Oct 30 21:59:21.476900 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:21.476867 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4797e46e-7da8-46f8-8b8f-77109e06e63f" path="/var/lib/kubelet/pods/4797e46e-7da8-46f8-8b8f-77109e06e63f/volumes" Oct 30 21:59:21.477390 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:21.477295 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" path="/var/lib/kubelet/pods/a52b3140-9db4-4f51-84d4-ce72106b5bc7/volumes" Oct 30 21:59:26.997434 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.997401 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 21:59:26.998013 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.997998 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" containerName="payload-pre-processing" Oct 30 21:59:26.998058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.998015 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" containerName="payload-pre-processing" Oct 30 21:59:26.998058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.998031 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4797e46e-7da8-46f8-8b8f-77109e06e63f" containerName="payload-processing" Oct 30 21:59:26.998058 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.998039 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4797e46e-7da8-46f8-8b8f-77109e06e63f" containerName="payload-processing" Oct 30 21:59:26.998146 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.998140 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52b3140-9db4-4f51-84d4-ce72106b5bc7" containerName="payload-pre-processing" Oct 30 21:59:26.998183 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:26.998148 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4797e46e-7da8-46f8-8b8f-77109e06e63f" containerName="payload-processing" Oct 30 21:59:27.003095 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.003076 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.055362 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.055331 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71a3d788-serving-cert\"" Oct 30 21:59:27.091882 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.091852 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.092057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.092031 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.092057 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.092051 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nhc\" (UniqueName: \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.092147 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.092086 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.109038 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.109008 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 21:59:27.193686 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.193641 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.193871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.193697 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nhc\" (UniqueName: \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.193871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.193759 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.193871 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.193836 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.194031 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:27.193954 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-serving-cert: secret "maas-api-e2e-mt-71a3d788-serving-cert" not found Oct 30 21:59:27.194087 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:27.194047 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls podName:c56444bc-4171-45a4-827d-f38176df0362 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:27.69401583 +0000 UTC m=+2192.368062348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls") pod "maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" (UID: "c56444bc-4171-45a4-827d-f38176df0362") : secret "maas-api-e2e-mt-71a3d788-serving-cert" not found Oct 30 21:59:27.194311 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.194292 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.196698 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.196674 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.303744 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.296047 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nhc\" (UniqueName: \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.708710 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.708668 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.711318 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.711288 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") pod \"maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:27.917080 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:27.917044 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:28.298494 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:28.298464 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 21:59:28.299384 ip-10-0-134-53 kubenswrapper[4525]: W1030 21:59:28.299358 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56444bc_4171_45a4_827d_f38176df0362.slice/crio-d81ad0f6779441a56380cd62cc0170037b6014ef5a07346d8ec8b3784c2cf84f WatchSource:0}: Error finding container d81ad0f6779441a56380cd62cc0170037b6014ef5a07346d8ec8b3784c2cf84f: Status 404 returned error can't find the container with id d81ad0f6779441a56380cd62cc0170037b6014ef5a07346d8ec8b3784c2cf84f Oct 30 21:59:28.502682 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:28.502638 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" event={"ID":"c56444bc-4171-45a4-827d-f38176df0362","Type":"ContainerStarted","Data":"d81ad0f6779441a56380cd62cc0170037b6014ef5a07346d8ec8b3784c2cf84f"} Oct 30 21:59:29.510762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:29.510663 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" event={"ID":"c56444bc-4171-45a4-827d-f38176df0362","Type":"ContainerStarted","Data":"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec"} Oct 30 21:59:29.510762 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:29.510732 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:29.629205 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:29.629152 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" podStartSLOduration=2.6605907330000003 podStartE2EDuration="3.629137406s" totalImagesPullingTime="968.546673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:29.628036224 +0000 UTC m=+2194.302082760" watchObservedRunningTime="2026-10-30 21:59:29.629137406 +0000 UTC m=+2194.303183941" Oct 30 21:59:30.309651 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:30.309600 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 21:59:35.396907 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:35.396864 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 21:59:35.522746 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:35.522716 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 21:59:56.062671 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.062636 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:59:56.063206 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.062871 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" containerID="cri-o://37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9" gracePeriod=30 Oct 30 21:59:56.129839 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.129786 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": dial tcp 10.133.0.128:8443: connect: connection refused" Oct 30 21:59:56.381730 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.381699 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:59:56.432432 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.432392 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\") pod \"6de079fd-0bbd-4b58-864a-8a1d461245f5\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " Oct 30 21:59:56.432662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.432445 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") pod \"6de079fd-0bbd-4b58-864a-8a1d461245f5\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " Oct 30 21:59:56.432662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.432513 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\" (UniqueName: \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\") pod \"6de079fd-0bbd-4b58-864a-8a1d461245f5\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " Oct 30 21:59:56.432662 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.432562 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\") pod \"6de079fd-0bbd-4b58-864a-8a1d461245f5\" (UID: \"6de079fd-0bbd-4b58-864a-8a1d461245f5\") " Oct 30 21:59:56.432858 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.432833 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca" pod "6de079fd-0bbd-4b58-864a-8a1d461245f5" (UID: "6de079fd-0bbd-4b58-864a-8a1d461245f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:56.433150 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.433129 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de079fd-0bbd-4b58-864a-8a1d461245f5-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.434944 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.434911 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs" pod "6de079fd-0bbd-4b58-864a-8a1d461245f5" (UID: "6de079fd-0bbd-4b58-864a-8a1d461245f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:56.435101 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.435079 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls" pod "6de079fd-0bbd-4b58-864a-8a1d461245f5" (UID: "6de079fd-0bbd-4b58-864a-8a1d461245f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:56.435147 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.435124 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4" pod "6de079fd-0bbd-4b58-864a-8a1d461245f5" (UID: "6de079fd-0bbd-4b58-864a-8a1d461245f5"). InnerVolumeSpecName "kube-api-access-lvdv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:56.536680 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.536646 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.536916 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.536897 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdv4\" (UniqueName: \"kubernetes.io/projected/6de079fd-0bbd-4b58-864a-8a1d461245f5-kube-api-access-lvdv4\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.537010 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.536998 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6de079fd-0bbd-4b58-864a-8a1d461245f5-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:56.724559 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.724514 4525 generic.go:356] "Generic (PLEG): container finished" podID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerID="37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9" exitCode=0 Oct 30 21:59:56.724790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.724584 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" Oct 30 21:59:56.724790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.724601 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerDied","Data":"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9"} Oct 30 21:59:56.724790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.724698 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l" event={"ID":"6de079fd-0bbd-4b58-864a-8a1d461245f5","Type":"ContainerDied","Data":"7ccb5e72754c53bcbd85bb64d83850b317a0492fa8058b870ea031f9130878fa"} Oct 30 21:59:56.724790 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.724725 4525 scope.go:122] "RemoveContainer" containerID="37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9" Oct 30 21:59:56.736783 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.736700 4525 scope.go:122] "RemoveContainer" containerID="e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c" Oct 30 21:59:56.745990 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.745969 4525 scope.go:122] "RemoveContainer" containerID="37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9" Oct 30 21:59:56.746279 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:56.746262 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9\": container with ID starting with 37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9 not found: ID does not exist" containerID="37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9" Oct 30 21:59:56.746355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.746287 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9"} err="failed to get container status \"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9\": rpc error: code = NotFound desc = could not find container \"37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9\": container with ID starting with 37fd6714fdf5112159382ef639923af9eb1077b4ddd09cf91c9d86adfc57b7e9 not found: ID does not exist" Oct 30 21:59:56.746355 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.746304 4525 scope.go:122] "RemoveContainer" containerID="e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c" Oct 30 21:59:56.746556 ip-10-0-134-53 kubenswrapper[4525]: E1030 21:59:56.746538 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c\": container with ID starting with e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c not found: ID does not exist" containerID="e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c" Oct 30 21:59:56.746635 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.746560 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c"} err="failed to get container status \"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c\": rpc error: code = NotFound desc = could not find container \"e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c\": container with ID starting with e86ef456347934fbd2ba7618f868d7535754c7ac477082601c79ee0fd1702f6c not found: ID does not exist" Oct 30 21:59:56.890875 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.890792 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:59:56.940971 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:56.940937 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5bfa09-6f48c47cdf-6zg4l"] Oct 30 21:59:57.474517 ip-10-0-134-53 kubenswrapper[4525]: I1030 21:59:57.474485 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" path="/var/lib/kubelet/pods/6de079fd-0bbd-4b58-864a-8a1d461245f5/volumes" Oct 30 22:00:00.204990 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.204954 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:00.205820 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.205799 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.205952 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.205821 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.206041 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.206028 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.206097 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.206056 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.211450 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.211428 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.228025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.227998 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:00.228579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.228566 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.228664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.228580 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de079fd-0bbd-4b58-864a-8a1d461245f5" containerName="maas-api" Oct 30 22:00:00.231957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.231935 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.290376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.290333 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.290589 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.290427 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqvn\" (UniqueName: \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.290589 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.290476 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.290589 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.290507 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88n5\" (UniqueName: \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.325216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.325180 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:00.331836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.331800 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z2hc6\"" Oct 30 22:00:00.380072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.380041 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:00.392202 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.392167 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.392367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.392223 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqvn\" (UniqueName: \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.392367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.392277 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.392367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.392316 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88n5\" (UniqueName: \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.393016 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.392995 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.393150 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.393129 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.454147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.454112 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88n5\" (UniqueName: \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\") pod \"maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.454309 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.454196 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqvn\" (UniqueName: \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.526044 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.525971 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:00.546584 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.546548 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:00.910993 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.910961 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:00.912244 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:00.912212 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa76c1d_f7b5_44c4_b19e_29fb64107379.slice/crio-d9c9084c4f5688c4b0b075b3d8d03dd65b19913a8754e8239c751a8b1a6e4c72 WatchSource:0}: Error finding container d9c9084c4f5688c4b0b075b3d8d03dd65b19913a8754e8239c751a8b1a6e4c72: Status 404 returned error can't find the container with id d9c9084c4f5688c4b0b075b3d8d03dd65b19913a8754e8239c751a8b1a6e4c72 Oct 30 22:00:00.960884 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:00.960857 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:00.962131 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:00.962100 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15c96b8_85d6_4616_92cf_c3b694bfddc1.slice/crio-bfa8632e047610631d8e64debfef99fa1cd3e871706afa743701a3c16d1d3d1a WatchSource:0}: Error finding container bfa8632e047610631d8e64debfef99fa1cd3e871706afa743701a3c16d1d3d1a: Status 404 returned error can't find the container with id bfa8632e047610631d8e64debfef99fa1cd3e871706afa743701a3c16d1d3d1a Oct 30 22:00:01.758781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.758748 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerStarted","Data":"54d05c132ba72b8fb2a432f0c3f842fb1a3698e574ebd3dfbbe422bc65740f41"} Oct 30 22:00:01.759358 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.758804 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerStarted","Data":"d9c9084c4f5688c4b0b075b3d8d03dd65b19913a8754e8239c751a8b1a6e4c72"} Oct 30 22:00:01.760715 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.760681 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerStarted","Data":"3be5928083574fca0fc905a287998822bc02adf2eafdbe3df89466f1171cb3ff"} Oct 30 22:00:01.760842 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.760738 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerStarted","Data":"bfa8632e047610631d8e64debfef99fa1cd3e871706afa743701a3c16d1d3d1a"} Oct 30 22:00:01.876057 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.876000 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" podStartSLOduration=1.8759855079999999 podStartE2EDuration="1.875985508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.8744975 +0000 UTC m=+2226.548544037" watchObservedRunningTime="2026-10-30 22:00:01.875985508 +0000 UTC m=+2226.550032044" Oct 30 22:00:01.991794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:01.991742 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" podStartSLOduration=1.991728688 podStartE2EDuration="1.991728688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.991278573 +0000 UTC m=+2226.665325107" watchObservedRunningTime="2026-10-30 22:00:01.991728688 +0000 UTC m=+2226.665775225" Oct 30 22:00:02.770017 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.769981 4525 generic.go:356] "Generic (PLEG): container finished" podID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerID="3be5928083574fca0fc905a287998822bc02adf2eafdbe3df89466f1171cb3ff" exitCode=60 Oct 30 22:00:02.770474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.770069 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerDied","Data":"3be5928083574fca0fc905a287998822bc02adf2eafdbe3df89466f1171cb3ff"} Oct 30 22:00:02.770474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.770394 4525 scope.go:122] "RemoveContainer" containerID="3be5928083574fca0fc905a287998822bc02adf2eafdbe3df89466f1171cb3ff" Oct 30 22:00:02.772094 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.772068 4525 generic.go:356] "Generic (PLEG): container finished" podID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerID="54d05c132ba72b8fb2a432f0c3f842fb1a3698e574ebd3dfbbe422bc65740f41" exitCode=60 Oct 30 22:00:02.772239 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.772135 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerDied","Data":"54d05c132ba72b8fb2a432f0c3f842fb1a3698e574ebd3dfbbe422bc65740f41"} Oct 30 22:00:02.772389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:02.772375 4525 scope.go:122] "RemoveContainer" containerID="54d05c132ba72b8fb2a432f0c3f842fb1a3698e574ebd3dfbbe422bc65740f41" Oct 30 22:00:03.780536 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.780439 4525 generic.go:356] "Generic (PLEG): container finished" podID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerID="89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98" exitCode=60 Oct 30 22:00:03.781074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.780532 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerDied","Data":"89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98"} Oct 30 22:00:03.781074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.780685 4525 scope.go:122] "RemoveContainer" containerID="54d05c132ba72b8fb2a432f0c3f842fb1a3698e574ebd3dfbbe422bc65740f41" Oct 30 22:00:03.781074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.780851 4525 scope.go:122] "RemoveContainer" containerID="89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98" Oct 30 22:00:03.781224 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:03.781186 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm_odh-ai-gateway-infra(6aa76c1d-f7b5-44c4-b19e-29fb64107379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" Oct 30 22:00:03.782723 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.782702 4525 generic.go:356] "Generic (PLEG): container finished" podID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerID="f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc" exitCode=60 Oct 30 22:00:03.782830 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.782760 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerDied","Data":"f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc"} Oct 30 22:00:03.783036 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.783022 4525 scope.go:122] "RemoveContainer" containerID="f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc" Oct 30 22:00:03.783315 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:03.783299 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w_odh-ai-gateway-infra(b15c96b8-85d6-4616-92cf-c3b694bfddc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" Oct 30 22:00:03.793413 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:03.793395 4525 scope.go:122] "RemoveContainer" containerID="3be5928083574fca0fc905a287998822bc02adf2eafdbe3df89466f1171cb3ff" Oct 30 22:00:04.790672 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:04.790647 4525 scope.go:122] "RemoveContainer" containerID="f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc" Oct 30 22:00:04.791153 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:04.791040 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w_odh-ai-gateway-infra(b15c96b8-85d6-4616-92cf-c3b694bfddc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" Oct 30 22:00:04.792350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:04.792329 4525 scope.go:122] "RemoveContainer" containerID="89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98" Oct 30 22:00:04.792697 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:04.792677 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm_odh-ai-gateway-infra(6aa76c1d-f7b5-44c4-b19e-29fb64107379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" Oct 30 22:00:05.397571 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:05.397535 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:00:06.663191 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:06.663147 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 22:00:06.663698 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:06.663365 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" containerID="cri-o://801789cd191a5796a08db69d142b33245ae09b05bc4d0bead64528774f62b698" gracePeriod=30 Oct 30 22:00:06.666323 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:06.666299 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 22:00:06.666481 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:06.666463 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" containerID="cri-o://a351c39453b2e1b8ddd278914ab47406b27f397d8d93fe5e20809df056c7579a" gracePeriod=30 Oct 30 22:00:08.001900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.001861 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:00:08.007657 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.007596 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.161631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.161493 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ae1bd5-openshift-default-dockercfg-ffdj4\"" Oct 30 22:00:08.168318 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.167524 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:00:08.168880 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.168843 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169011 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.168926 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169006 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169057 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169166 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtws\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169246 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169309 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169273 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169415 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169401 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.169461 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.169438 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.271442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.271353 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.271753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.271730 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.271921 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.271903 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272120 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272105 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtws\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272276 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272262 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272367 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272645 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272778 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272764 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.272937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.272921 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.277160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.274060 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.277160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.274282 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.277160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.274301 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.277160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.277120 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.278092 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.278073 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.279045 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.279010 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.280034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.280020 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.342865 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.342776 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.348235 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.348157 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtws\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\") pod \"e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:08.625658 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:08.625349 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:09.059574 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.059493 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:00:09.095552 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:09.095523 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b544a7d_7df3_4799_841c_f5069200e931.slice/crio-5bbdb4640099f43545c36fe265e6824f7f477a8ab4044e18b4a782e02e3745d3 WatchSource:0}: Error finding container 5bbdb4640099f43545c36fe265e6824f7f477a8ab4044e18b4a782e02e3745d3: Status 404 returned error can't find the container with id 5bbdb4640099f43545c36fe265e6824f7f477a8ab4044e18b4a782e02e3745d3 Oct 30 22:00:09.098166 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.098133 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:09.098234 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.098221 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:09.098273 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.098256 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:09.883282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.883241 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerStarted","Data":"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab"} Oct 30 22:00:09.883620 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:09.883568 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerStarted","Data":"5bbdb4640099f43545c36fe265e6824f7f477a8ab4044e18b4a782e02e3745d3"} Oct 30 22:00:10.058757 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:10.058703 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podStartSLOduration=3.058688379 podStartE2EDuration="3.058688379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:10.055962385 +0000 UTC m=+2234.730008924" watchObservedRunningTime="2026-10-30 22:00:10.058688379 +0000 UTC m=+2234.732734914" Oct 30 22:00:10.626669 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:10.626633 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:10.632924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:10.632901 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:10.890407 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:10.890308 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:10.891429 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:10.891409 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:00:11.262329 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.262247 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:00:11.672700 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.672669 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:00:11.681876 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.681114 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.743094 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.743061 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:00:11.793264 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.793238 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e6dcc0-openshift-default-dockercfg-h46cd\"" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835155 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835245 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835307 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835338 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmwk\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835513 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835562 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835624 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835656 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.839633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.835719 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.900900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.900871 4525 generic.go:356] "Generic (PLEG): container finished" podID="460e245c-198d-4299-b681-bfb5d17a2053" containerID="801789cd191a5796a08db69d142b33245ae09b05bc4d0bead64528774f62b698" exitCode=0 Oct 30 22:00:11.901075 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.900937 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" event={"ID":"460e245c-198d-4299-b681-bfb5d17a2053","Type":"ContainerDied","Data":"801789cd191a5796a08db69d142b33245ae09b05bc4d0bead64528774f62b698"} Oct 30 22:00:11.902744 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.902715 4525 generic.go:356] "Generic (PLEG): container finished" podID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerID="a351c39453b2e1b8ddd278914ab47406b27f397d8d93fe5e20809df056c7579a" exitCode=0 Oct 30 22:00:11.903000 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.902957 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" event={"ID":"f181ba03-5400-431d-b981-937b8c1e1cc7","Type":"ContainerDied","Data":"a351c39453b2e1b8ddd278914ab47406b27f397d8d93fe5e20809df056c7579a"} Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975014 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975255 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975354 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975440 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975471 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmwk\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975565 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975629 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975716 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.975755 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.977524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.976315 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.978502 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.977969 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.993382 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.983520 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.993382 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.993326 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.993659 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.993480 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:11.999025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:11.998996 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:12.001368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.001306 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:12.047297 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.047260 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmwk\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:12.051309 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.051282 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\") pod \"e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:12.056784 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.056761 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 22:00:12.076262 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076232 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076359 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076386 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076500 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076554 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076583 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076789 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076630 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076789 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076649 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.076789 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.076705 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\") pod \"460e245c-198d-4299-b681-bfb5d17a2053\" (UID: \"460e245c-198d-4299-b681-bfb5d17a2053\") " Oct 30 22:00:12.078113 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.078087 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.078353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.078330 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.078461 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.078416 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.078742 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.078717 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:12.078940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.078917 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.080866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.080812 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.080866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.080820 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.080866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.080848 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:12.081038 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.080862 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj" pod "460e245c-198d-4299-b681-bfb5d17a2053" (UID: "460e245c-198d-4299-b681-bfb5d17a2053"). InnerVolumeSpecName "kube-api-access-nrdrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.110289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.110264 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 22:00:12.178469 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178425 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178514 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178549 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178656 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178688 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178965 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178741 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178965 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178774 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178965 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178812 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.178965 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.178866 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\") pod \"f181ba03-5400-431d-b981-937b8c1e1cc7\" (UID: \"f181ba03-5400-431d-b981-937b8c1e1cc7\") " Oct 30 22:00:12.179387 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179211 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.179495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179422 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.179563 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179534 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179563 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179557 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdrj\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-kube-api-access-nrdrj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179695 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179573 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179751 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179579 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:12.179751 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179697 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.179751 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179586 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460e245c-198d-4299-b681-bfb5d17a2053-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179751 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179736 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179752 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460e245c-198d-4299-b681-bfb5d17a2053-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179767 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179782 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179796 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460e245c-198d-4299-b681-bfb5d17a2053-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179811 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460e245c-198d-4299-b681-bfb5d17a2053-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.179974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.179960 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.182173 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.182143 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:12.182173 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.182148 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:12.182352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.182330 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:12.182506 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.182483 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "kube-api-access-t5xh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.182623 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.182551 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token" pod "f181ba03-5400-431d-b981-937b8c1e1cc7" (UID: "f181ba03-5400-431d-b981-937b8c1e1cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:12.281755 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281723 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281755 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281757 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xh9\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-kube-api-access-t5xh9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281776 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281789 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281803 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281817 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281831 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f181ba03-5400-431d-b981-937b8c1e1cc7-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.281983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.281843 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f181ba03-5400-431d-b981-937b8c1e1cc7-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:12.630951 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.630922 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:00:12.631634 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:12.631589 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c1417d_e609_42c9_a9f7_afde6ed45e5c.slice/crio-e099ad11708b9d57f59bbf1383317b15ce599f7297bdbf8d67e359088d83c21a WatchSource:0}: Error finding container e099ad11708b9d57f59bbf1383317b15ce599f7297bdbf8d67e359088d83c21a: Status 404 returned error can't find the container with id e099ad11708b9d57f59bbf1383317b15ce599f7297bdbf8d67e359088d83c21a Oct 30 22:00:12.634442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.634407 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:12.634545 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.634491 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:12.634545 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.634531 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:12.914683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.913875 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" event={"ID":"460e245c-198d-4299-b681-bfb5d17a2053","Type":"ContainerDied","Data":"7e1185d17ec81945191c7bc85e3389fdac404ac11b7c00686811e1c41b73dcca"} Oct 30 22:00:12.914683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.913933 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww" Oct 30 22:00:12.914683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.913954 4525 scope.go:122] "RemoveContainer" containerID="801789cd191a5796a08db69d142b33245ae09b05bc4d0bead64528774f62b698" Oct 30 22:00:12.931114 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.930994 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" Oct 30 22:00:12.931114 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.931041 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np" event={"ID":"f181ba03-5400-431d-b981-937b8c1e1cc7","Type":"ContainerDied","Data":"178c69b24c22ef3021285ba1e0ae90b410bb789c362edc6dec6f718a33b22d33"} Oct 30 22:00:12.938406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.938365 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerStarted","Data":"6b33408d899402aaa07623e8f87042ae60ff056cdb5c3c407862efd27bf47849"} Oct 30 22:00:12.938721 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.938651 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerStarted","Data":"e099ad11708b9d57f59bbf1383317b15ce599f7297bdbf8d67e359088d83c21a"} Oct 30 22:00:12.979639 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:12.976976 4525 scope.go:122] "RemoveContainer" containerID="a351c39453b2e1b8ddd278914ab47406b27f397d8d93fe5e20809df056c7579a" Oct 30 22:00:13.120827 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.120793 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 22:00:13.127343 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.127308 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-qp8np"] Oct 30 22:00:13.216738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.198953 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:13.216738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.207780 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:00:13.216738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.207837 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:00:13.337636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.331911 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 22:00:13.337636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.336063 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5bfa09-openshift-default-5d9f9b794f-zdzww"] Oct 30 22:00:13.441635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.430438 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podStartSLOduration=2.430417265 podStartE2EDuration="2.430417265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:13.425160478 +0000 UTC m=+2238.099207014" watchObservedRunningTime="2026-10-30 22:00:13.430417265 +0000 UTC m=+2238.104463802" Oct 30 22:00:13.475003 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.474926 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460e245c-198d-4299-b681-bfb5d17a2053" path="/var/lib/kubelet/pods/460e245c-198d-4299-b681-bfb5d17a2053/volumes" Oct 30 22:00:13.475472 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:13.475458 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" path="/var/lib/kubelet/pods/f181ba03-5400-431d-b981-937b8c1e1cc7/volumes" Oct 30 22:00:14.188207 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:14.188180 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:14.969618 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:14.969565 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:14.973179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:14.973157 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:00:17.466599 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:17.466485 4525 scope.go:122] "RemoveContainer" containerID="89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98" Oct 30 22:00:17.994962 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:17.994931 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerStarted","Data":"5ef57f159f1e4f70ef28b5d78893b2757e362f997ded7af1f0c8e85b41a36d21"} Oct 30 22:00:18.467341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:18.467311 4525 scope.go:122] "RemoveContainer" containerID="f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc" Oct 30 22:00:19.004006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.003976 4525 generic.go:356] "Generic (PLEG): container finished" podID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerID="5ef57f159f1e4f70ef28b5d78893b2757e362f997ded7af1f0c8e85b41a36d21" exitCode=60 Oct 30 22:00:19.004177 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.004052 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerDied","Data":"5ef57f159f1e4f70ef28b5d78893b2757e362f997ded7af1f0c8e85b41a36d21"} Oct 30 22:00:19.004177 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.004150 4525 scope.go:122] "RemoveContainer" containerID="89a9ab8b24fcb12bb640f79c180663cceb2f0e3490221aaaf015ade6a7409f98" Oct 30 22:00:19.004646 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.004575 4525 scope.go:122] "RemoveContainer" containerID="5ef57f159f1e4f70ef28b5d78893b2757e362f997ded7af1f0c8e85b41a36d21" Oct 30 22:00:19.005005 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:19.004981 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm_odh-ai-gateway-infra(6aa76c1d-f7b5-44c4-b19e-29fb64107379)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" Oct 30 22:00:19.006221 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.006193 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerStarted","Data":"ab5eecc2d1a6c2c22b6156a7a14412fe0f64cd7e35c41e4948acad831372e9b6"} Oct 30 22:00:19.157699 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:19.157667 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:20.014365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.014330 4525 generic.go:356] "Generic (PLEG): container finished" podID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerID="ab5eecc2d1a6c2c22b6156a7a14412fe0f64cd7e35c41e4948acad831372e9b6" exitCode=60 Oct 30 22:00:20.014934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.014404 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerDied","Data":"ab5eecc2d1a6c2c22b6156a7a14412fe0f64cd7e35c41e4948acad831372e9b6"} Oct 30 22:00:20.014934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.014481 4525 scope.go:122] "RemoveContainer" containerID="f8f4ea2f9b58e19875d8a500afd81e6b998de2d32f1dcd511ff14b49d74122cc" Oct 30 22:00:20.015050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.014941 4525 scope.go:122] "RemoveContainer" containerID="ab5eecc2d1a6c2c22b6156a7a14412fe0f64cd7e35c41e4948acad831372e9b6" Oct 30 22:00:20.015280 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:20.015253 4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w_odh-ai-gateway-infra(b15c96b8-85d6-4616-92cf-c3b694bfddc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.203214 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204182 4525 cpu_manager.go:414] "containerMap: removing container" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204203 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204271 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204280 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204426 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f181ba03-5400-431d-b981-937b8c1e1cc7" containerName="istio-proxy" Oct 30 22:00:20.205280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.204448 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="460e245c-198d-4299-b681-bfb5d17a2053" containerName="istio-proxy" Oct 30 22:00:20.215037 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.214301 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.271390 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.271325 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e6dcc0\"" Oct 30 22:00:20.271540 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.271505 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e6dcc0-dockercfg-gp4wd\"" Oct 30 22:00:20.272801 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.272763 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:00:20.278197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.278170 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:20.285456 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.285427 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:20.336649 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.335273 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.336649 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.335362 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbghm\" (UniqueName: \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.434809 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435713 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435730 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435742 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435748 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435760 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435767 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435921 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435931 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.438633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.435939 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" containerName="cleanup" Oct 30 22:00:20.440108 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.440074 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\") pod \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " Oct 30 22:00:20.440296 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.440280 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\" (UniqueName: \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\") pod \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\" (UID: \"6aa76c1d-f7b5-44c4-b19e-29fb64107379\") " Oct 30 22:00:20.441037 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.441005 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.441223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.441208 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbghm\" (UniqueName: \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.446367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.442316 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.447504 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.447467 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca" pod "6aa76c1d-f7b5-44c4-b19e-29fb64107379" (UID: "6aa76c1d-f7b5-44c4-b19e-29fb64107379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.455618 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.455566 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:00:20.455805 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.455716 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.460859 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.460813 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5" pod "6aa76c1d-f7b5-44c4-b19e-29fb64107379" (UID: "6aa76c1d-f7b5-44c4-b19e-29fb64107379"). InnerVolumeSpecName "kube-api-access-n88n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:20.503641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.503574 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbghm\" (UniqueName: \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\") pod \"payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.535231 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.535075 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:20.543073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.543042 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62m2\" (UniqueName: \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.543249 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.543229 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.544628 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.543782 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aa76c1d-f7b5-44c4-b19e-29fb64107379-maas-api-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.544628 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.543809 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88n5\" (UniqueName: \"kubernetes.io/projected/6aa76c1d-f7b5-44c4-b19e-29fb64107379-kube-api-access-n88n5\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:20.646719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.644968 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62m2\" (UniqueName: \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.646719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.645171 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.646719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.646348 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.708122 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.708087 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62m2\" (UniqueName: \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\") pod \"payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.790997 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.790904 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:20.928896 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:20.928511 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:00:20.931383 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:20.931357 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508779a4_73f5_48e8_869d_4e54f50fef0a.slice/crio-8880154d33a82bd79334087a657bcebb2c6a3d08e046015a8bf7e4414f38c7e7 WatchSource:0}: Error finding container 8880154d33a82bd79334087a657bcebb2c6a3d08e046015a8bf7e4414f38c7e7: Status 404 returned error can't find the container with id 8880154d33a82bd79334087a657bcebb2c6a3d08e046015a8bf7e4414f38c7e7 Oct 30 22:00:21.024458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.024426 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" event={"ID":"6aa76c1d-f7b5-44c4-b19e-29fb64107379","Type":"ContainerDied","Data":"d9c9084c4f5688c4b0b075b3d8d03dd65b19913a8754e8239c751a8b1a6e4c72"} Oct 30 22:00:21.024943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.024438 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm" Oct 30 22:00:21.024943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.024506 4525 scope.go:122] "RemoveContainer" containerID="5ef57f159f1e4f70ef28b5d78893b2757e362f997ded7af1f0c8e85b41a36d21" Oct 30 22:00:21.026388 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.026360 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" event={"ID":"508779a4-73f5-48e8-869d-4e54f50fef0a","Type":"ContainerStarted","Data":"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3"} Oct 30 22:00:21.026509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.026415 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" event={"ID":"508779a4-73f5-48e8-869d-4e54f50fef0a","Type":"ContainerStarted","Data":"8880154d33a82bd79334087a657bcebb2c6a3d08e046015a8bf7e4414f38c7e7"} Oct 30 22:00:21.026509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.026439 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:21.162216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.162188 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:21.175757 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.175735 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:00:21.176457 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:21.176431 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987f3950_569e_47e4_96b7_995d0289b09a.slice/crio-f0bb7fa4c160942a473bd770f9dfef29691ac7cbacc8470fdbb530106034a74f WatchSource:0}: Error finding container f0bb7fa4c160942a473bd770f9dfef29691ac7cbacc8470fdbb530106034a74f: Status 404 returned error can't find the container with id f0bb7fa4c160942a473bd770f9dfef29691ac7cbacc8470fdbb530106034a74f Oct 30 22:00:21.203379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.203334 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" podStartSLOduration=1.203319891 podStartE2EDuration="1.203319891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:21.199368372 +0000 UTC m=+2245.873414919" watchObservedRunningTime="2026-10-30 22:00:21.203319891 +0000 UTC m=+2245.877366427" Oct 30 22:00:21.255089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.255047 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\" (UniqueName: \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\") pod \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " Oct 30 22:00:21.255870 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.255124 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\") pod \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\" (UID: \"b15c96b8-85d6-4616-92cf-c3b694bfddc1\") " Oct 30 22:00:21.256008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.255988 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca" pod "b15c96b8-85d6-4616-92cf-c3b694bfddc1" (UID: "b15c96b8-85d6-4616-92cf-c3b694bfddc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.259023 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.258985 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn" pod "b15c96b8-85d6-4616-92cf-c3b694bfddc1" (UID: "b15c96b8-85d6-4616-92cf-c3b694bfddc1"). InnerVolumeSpecName "kube-api-access-tmqvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.317198 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.317160 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:21.320384 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.320359 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d68156-29889960-g74pm"] Oct 30 22:00:21.363748 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.361051 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqvn\" (UniqueName: \"kubernetes.io/projected/b15c96b8-85d6-4616-92cf-c3b694bfddc1-kube-api-access-tmqvn\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.363748 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.361089 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b15c96b8-85d6-4616-92cf-c3b694bfddc1-maas-api-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.474986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:21.474956 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa76c1d-f7b5-44c4-b19e-29fb64107379" path="/var/lib/kubelet/pods/6aa76c1d-f7b5-44c4-b19e-29fb64107379/volumes" Oct 30 22:00:22.039980 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.039939 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" Oct 30 22:00:22.040440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.039940 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w" event={"ID":"b15c96b8-85d6-4616-92cf-c3b694bfddc1","Type":"ContainerDied","Data":"bfa8632e047610631d8e64debfef99fa1cd3e871706afa743701a3c16d1d3d1a"} Oct 30 22:00:22.040440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.040122 4525 scope.go:122] "RemoveContainer" containerID="ab5eecc2d1a6c2c22b6156a7a14412fe0f64cd7e35c41e4948acad831372e9b6" Oct 30 22:00:22.043266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.043186 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" event={"ID":"987f3950-569e-47e4-96b7-995d0289b09a","Type":"ContainerStarted","Data":"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44"} Oct 30 22:00:22.043266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.043251 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" event={"ID":"987f3950-569e-47e4-96b7-995d0289b09a","Type":"ContainerStarted","Data":"f0bb7fa4c160942a473bd770f9dfef29691ac7cbacc8470fdbb530106034a74f"} Oct 30 22:00:22.043470 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.043366 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:22.213963 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.213910 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" podStartSLOduration=2.213895387 podStartE2EDuration="2.213895387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:22.211162167 +0000 UTC m=+2246.885208704" watchObservedRunningTime="2026-10-30 22:00:22.213895387 +0000 UTC m=+2246.887941922" Oct 30 22:00:22.346035 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.337795 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:22.346035 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:22.343086 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1a415f-29889960-xc69w"] Oct 30 22:00:23.475382 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.475350 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" path="/var/lib/kubelet/pods/b15c96b8-85d6-4616-92cf-c3b694bfddc1/volumes" Oct 30 22:00:23.566860 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.566827 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:00:23.567505 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567485 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.567548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567512 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.567548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567539 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.567548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567545 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.567688 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567682 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.567722 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.567693 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:23.572507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.572480 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693045 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693087 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693284 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693107 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8hl\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693284 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693271 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693307 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693392 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693380 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693436 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693424 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693472 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693446 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.693510 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.693489 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.784314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.784229 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:00:23.795255 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795218 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795412 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795290 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795412 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795358 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795412 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795403 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795555 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795427 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795555 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795484 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795642 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795622 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795655 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.795683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.795678 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8hl\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.796949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.796920 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.797080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.796926 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.797080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.796971 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.797256 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.797232 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.797329 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.797276 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.798988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.798959 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.799360 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.799338 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.857647 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.857597 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8hl\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.857647 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.857636 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-snq52\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:23.889509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:23.889479 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:24.333881 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:24.333845 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:00:24.334869 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:24.334846 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645f9f90_95b2_42e9_9808_c0db83424296.slice/crio-4f780a5b10f24cf097d0be1a1cff1d7069093b592944e508ed3c62e6da0a10cf WatchSource:0}: Error finding container 4f780a5b10f24cf097d0be1a1cff1d7069093b592944e508ed3c62e6da0a10cf: Status 404 returned error can't find the container with id 4f780a5b10f24cf097d0be1a1cff1d7069093b592944e508ed3c62e6da0a10cf Oct 30 22:00:24.337575 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:24.337541 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:24.337683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:24.337629 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:24.337683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:24.337662 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:25.065990 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.065954 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" event={"ID":"645f9f90-95b2-42e9-9808-c0db83424296","Type":"ContainerStarted","Data":"8ceba82ef7820c416340101889b48b5aae455baeb1e734a4d6264f0dc0a4a8e6"} Oct 30 22:00:25.066481 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.066010 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" event={"ID":"645f9f90-95b2-42e9-9808-c0db83424296","Type":"ContainerStarted","Data":"4f780a5b10f24cf097d0be1a1cff1d7069093b592944e508ed3c62e6da0a10cf"} Oct 30 22:00:25.265635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.265094 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podStartSLOduration=2.265077743 podStartE2EDuration="2.265077743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:25.198690498 +0000 UTC m=+2249.872737035" watchObservedRunningTime="2026-10-30 22:00:25.265077743 +0000 UTC m=+2249.939124278" Oct 30 22:00:25.270812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.270783 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:00:25.271464 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.271450 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:25.271507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.271467 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:25.271572 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.271562 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15c96b8-85d6-4616-92cf-c3b694bfddc1" containerName="cleanup" Oct 30 22:00:25.275032 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.275016 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.334686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.334440 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-98d38f-dockercfg-bp69m\"" Oct 30 22:00:25.338635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.335233 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-98d38f\"" Oct 30 22:00:25.518932 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.518528 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.518932 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.518662 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk8d\" (UniqueName: \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.532591 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.532558 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:00:25.532883 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.532867 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:00:25.564635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.564572 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.571852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.571802 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:00:25.620474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.620443 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk8d\" (UniqueName: \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.620992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.620797 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md49p\" (UniqueName: \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.620992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.620859 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.621192 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.621062 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.621527 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.621504 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.682956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.682926 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk8d\" (UniqueName: \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\") pod \"payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.722283 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.722244 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.722468 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.722454 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md49p\" (UniqueName: \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.722986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.722968 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.783429 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.783401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md49p\" (UniqueName: \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\") pod \"payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.885684 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.883769 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:25.911146 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.891959 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:25.911146 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.892419 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:25.911146 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.897105 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.911146 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:25.897157 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:26.474745 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:26.474661 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:00:26.513710 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:26.513667 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde248853_e91d_4c1d_b5d2_ff05662c6d13.slice/crio-fec091a17b699e09dace7598ae95ae0475876465010f19dd270632d5da8dfe41 WatchSource:0}: Error finding container fec091a17b699e09dace7598ae95ae0475876465010f19dd270632d5da8dfe41: Status 404 returned error can't find the container with id fec091a17b699e09dace7598ae95ae0475876465010f19dd270632d5da8dfe41 Oct 30 22:00:26.526754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:26.526729 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:00:26.546128 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:26.546095 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5860e9c8_31f9_4bbd_9ddc_cfb4f48014a8.slice/crio-fd5a10d2f44e24b344a80fe6285725a85f610cf5a4c280647e29511eff52f7c2 WatchSource:0}: Error finding container fd5a10d2f44e24b344a80fe6285725a85f610cf5a4c280647e29511eff52f7c2: Status 404 returned error can't find the container with id fd5a10d2f44e24b344a80fe6285725a85f610cf5a4c280647e29511eff52f7c2 Oct 30 22:00:26.890931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:26.890895 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.891088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:26.890948 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:27.113902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.113863 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" event={"ID":"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8","Type":"ContainerStarted","Data":"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf"} Oct 30 22:00:27.114105 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.113925 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" event={"ID":"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8","Type":"ContainerStarted","Data":"fd5a10d2f44e24b344a80fe6285725a85f610cf5a4c280647e29511eff52f7c2"} Oct 30 22:00:27.114105 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.113961 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:27.115671 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.115640 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" event={"ID":"de248853-e91d-4c1d-b5d2-ff05662c6d13","Type":"ContainerStarted","Data":"f35c62589a9bc47262dabfccea64e4769191746bfb73686aa6e81d5858e390e1"} Oct 30 22:00:27.115816 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.115692 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" event={"ID":"de248853-e91d-4c1d-b5d2-ff05662c6d13","Type":"ContainerStarted","Data":"fec091a17b699e09dace7598ae95ae0475876465010f19dd270632d5da8dfe41"} Oct 30 22:00:27.115816 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.115757 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:27.371637 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.370001 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" podStartSLOduration=2.369980921 podStartE2EDuration="2.369980921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:27.369218926 +0000 UTC m=+2252.043265470" watchObservedRunningTime="2026-10-30 22:00:27.369980921 +0000 UTC m=+2252.044027458" Oct 30 22:00:27.371637 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.371501 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" podStartSLOduration=2.371487683 podStartE2EDuration="2.371487683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:27.241897003 +0000 UTC m=+2251.915943542" watchObservedRunningTime="2026-10-30 22:00:27.371487683 +0000 UTC m=+2252.045534225" Oct 30 22:00:27.890581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.890548 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.891004 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:27.890650 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:28.890216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:28.890183 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:28.890405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:28.890242 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:29.890624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:29.890573 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:29.890984 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:29.890642 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:30.890809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:30.890760 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:00:30.891221 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:30.890835 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Oct 30 22:00:31.895163 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:31.895134 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:31.895658 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:31.895631 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:31.896393 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:31.896370 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:00:32.047282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:32.047257 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:00:33.055776 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:33.055748 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:00:35.099590 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.099527 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:00:35.106698 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.106673 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.155447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.155390 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:00:35.200931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.200900 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ae1bd5\"" Oct 30 22:00:35.224743 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.224704 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.236948 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.236910 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzqj\" (UniqueName: \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.359353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.359253 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.359546 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.359437 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzqj\" (UniqueName: \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.360993 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.360960 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.398970 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:35.398926 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:00:35.421198 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.421171 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzqj\" (UniqueName: \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\") pod \"payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.424133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.424115 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:35.802451 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:35.802419 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:00:35.803598 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:35.803559 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f64dfc8_9934_478c_8317_8f301f0f1b2e.slice/crio-8d08797954f265f308de6a61c09825e1a1fa17bc9b1f38b081e0179a31ac045e WatchSource:0}: Error finding container 8d08797954f265f308de6a61c09825e1a1fa17bc9b1f38b081e0179a31ac045e: Status 404 returned error can't find the container with id 8d08797954f265f308de6a61c09825e1a1fa17bc9b1f38b081e0179a31ac045e Oct 30 22:00:36.217165 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:36.217125 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" event={"ID":"8f64dfc8-9934-478c-8317-8f301f0f1b2e","Type":"ContainerStarted","Data":"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea"} Oct 30 22:00:36.217628 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:36.217192 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" event={"ID":"8f64dfc8-9934-478c-8317-8f301f0f1b2e","Type":"ContainerStarted","Data":"8d08797954f265f308de6a61c09825e1a1fa17bc9b1f38b081e0179a31ac045e"} Oct 30 22:00:36.217628 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:36.217236 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:36.337074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:36.337014 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" podStartSLOduration=1.336998045 podStartE2EDuration="1.336998045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:36.333822601 +0000 UTC m=+2261.007869137" watchObservedRunningTime="2026-10-30 22:00:36.336998045 +0000 UTC m=+2261.011044580" Oct 30 22:00:38.127558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.127518 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:00:38.128089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.127595 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:00:38.693720 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.693685 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:00:38.698343 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.698324 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721559 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721524 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721683 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721745 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721818 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721777 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721914 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721901 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.721963 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.721949 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.722050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.722036 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.722096 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.722083 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhxl\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.722131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.722118 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.806726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.806688 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:00:38.822986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.822943 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.822986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.822986 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823016 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhxl\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823036 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823140 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823225 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823251 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823552 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823285 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823552 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823423 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823552 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823425 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823756 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823558 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823815 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823757 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823815 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823802 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.823924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.823841 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.826188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.826158 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.826331 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.826201 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.885128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.885096 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:38.885297 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:38.885211 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhxl\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\") pod \"e2e-mt-71a3d788-openshift-default-778558588f-c8mwj\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:39.013073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:39.012979 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:39.450906 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:39.450878 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:00:39.451759 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:39.451733 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bd8d19_7d8c_4c63_98cb_acdf3a4c59d4.slice/crio-9329766eaa46278f069d00686775178db13d5226c245cc713f7a7b27007ede39 WatchSource:0}: Error finding container 9329766eaa46278f069d00686775178db13d5226c245cc713f7a7b27007ede39: Status 404 returned error can't find the container with id 9329766eaa46278f069d00686775178db13d5226c245cc713f7a7b27007ede39 Oct 30 22:00:39.454412 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:39.454376 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:39.454531 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:39.454458 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:39.454531 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:39.454505 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:40.262575 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:40.262528 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" event={"ID":"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4","Type":"ContainerStarted","Data":"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa"} Oct 30 22:00:40.262810 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:40.262670 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" event={"ID":"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4","Type":"ContainerStarted","Data":"9329766eaa46278f069d00686775178db13d5226c245cc713f7a7b27007ede39"} Oct 30 22:00:40.455893 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:40.455836 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podStartSLOduration=2.45581907 podStartE2EDuration="2.45581907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:40.434377521 +0000 UTC m=+2265.108424059" watchObservedRunningTime="2026-10-30 22:00:40.45581907 +0000 UTC m=+2265.129865620" Oct 30 22:00:41.013843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.013808 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:41.015061 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.015043 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:00:41.015120 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.015088 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:00:41.474723 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.474690 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:00:41.479158 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.479134 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.553200 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.553170 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:00:41.579631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.579580 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ea1f0b-openshift-default-dockercfg-8qsnz\"" Oct 30 22:00:41.597778 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.597739 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.602189 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.602149 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.612793 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.612739 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613030 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613206 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613364 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613348 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613567 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613550 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2m4\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613813 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613768 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.613977 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.613955 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850422 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850504 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850560 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850585 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850708 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850738 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850810 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2m4\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850849 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.850885 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.857737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.851409 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.924292 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.924248 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.924586 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.924561 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.925481 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.925215 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.941646 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.940628 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.971208 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.971137 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.974566 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.974533 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.990792 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.990752 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:41.991695 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:41.991669 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2m4\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\") pod \"e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:42.013905 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.013872 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:00:42.014058 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.013930 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:00:42.100185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.100109 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:42.658619 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.658561 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:00:42.670679 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:42.670649 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5327b3bf_027c_43b3_a35e_3ed832f71182.slice/crio-836646ac05fb8d81fdb9a21c38ffedbd2af30454423016f8265874e3e72d10eb WatchSource:0}: Error finding container 836646ac05fb8d81fdb9a21c38ffedbd2af30454423016f8265874e3e72d10eb: Status 404 returned error can't find the container with id 836646ac05fb8d81fdb9a21c38ffedbd2af30454423016f8265874e3e72d10eb Oct 30 22:00:42.673456 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.673421 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:42.673526 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.673495 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:42.673568 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:42.673525 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:43.014110 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:43.014025 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:00:43.014110 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:43.014081 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:00:43.300939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:43.300847 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerStarted","Data":"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87"} Oct 30 22:00:43.300939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:43.300935 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerStarted","Data":"836646ac05fb8d81fdb9a21c38ffedbd2af30454423016f8265874e3e72d10eb"} Oct 30 22:00:43.424379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:43.424323 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podStartSLOduration=2.424308785 podStartE2EDuration="2.424308785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:43.423210002 +0000 UTC m=+2268.097256539" watchObservedRunningTime="2026-10-30 22:00:43.424308785 +0000 UTC m=+2268.098355382" Oct 30 22:00:44.014174 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.014141 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:00:44.014618 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.014201 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:00:44.101774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.101741 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:44.107763 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.107736 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:44.308190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.308109 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:44.309181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:44.309158 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:00:45.013468 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:45.013431 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 22:00:45.013671 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:45.013520 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 22:00:46.566292 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:46.566253 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:46.567311 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:46.567283 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:46.568243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:46.568224 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:00:47.227157 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:47.227131 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:00:54.082460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:54.082427 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:00:55.965105 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:55.965072 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7088-d79e1a"] Oct 30 22:00:55.970590 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:55.970571 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:00:56.013438 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.013399 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9ct\" (UniqueName: \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\") pod \"test-curl-7088-d79e1a\" (UID: \"fe20ca0a-deae-4cff-b2c3-60717bafa4f0\") " pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:00:56.023619 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.023573 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7088-d79e1a"] Oct 30 22:00:56.068961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.068935 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncw9l\"" Oct 30 22:00:56.114725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.114687 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9ct\" (UniqueName: \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\") pod \"test-curl-7088-d79e1a\" (UID: \"fe20ca0a-deae-4cff-b2c3-60717bafa4f0\") " pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:00:56.186645 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.186585 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9ct\" (UniqueName: \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\") pod \"test-curl-7088-d79e1a\" (UID: \"fe20ca0a-deae-4cff-b2c3-60717bafa4f0\") " pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:00:56.283553 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.283475 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:00:56.666480 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:56.666451 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7088-d79e1a"] Oct 30 22:00:56.667413 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:00:56.667385 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe20ca0a_deae_4cff_b2c3_60717bafa4f0.slice/crio-665b4043d9145364f23b7bd1102f771d594c08181efb9d158d2b6ef66fbcacb0 WatchSource:0}: Error finding container 665b4043d9145364f23b7bd1102f771d594c08181efb9d158d2b6ef66fbcacb0: Status 404 returned error can't find the container with id 665b4043d9145364f23b7bd1102f771d594c08181efb9d158d2b6ef66fbcacb0 Oct 30 22:00:57.395546 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:57.395512 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7088-d79e1a" event={"ID":"fe20ca0a-deae-4cff-b2c3-60717bafa4f0","Type":"ContainerStarted","Data":"d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0"} Oct 30 22:00:57.395982 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:57.395555 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7088-d79e1a" event={"ID":"fe20ca0a-deae-4cff-b2c3-60717bafa4f0","Type":"ContainerStarted","Data":"665b4043d9145364f23b7bd1102f771d594c08181efb9d158d2b6ef66fbcacb0"} Oct 30 22:00:57.512245 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:57.512189 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7088-d79e1a" podStartSLOduration=2.5121731130000002 podStartE2EDuration="2.512173113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:57.509066682 +0000 UTC m=+2282.183113224" watchObservedRunningTime="2026-10-30 22:00:57.512173113 +0000 UTC m=+2282.186219648" Oct 30 22:00:59.113559 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:59.113524 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7088-d79e1a"] Oct 30 22:00:59.119985 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:59.119958 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7088-d79e1a"] Oct 30 22:00:59.408773 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:00:59.408737 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7088-d79e1a" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" containerName="test-curl-7088-d79e1a" containerID="cri-o://d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0" gracePeriod=2 Oct 30 22:00:59.460159 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:00:59.460119 4525 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7088-d79e1a\" is forbidden: User \"system:node:ip-10-0-134-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-53.us-west-1.compute.internal' and this object" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:01:00.068151 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.067947 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:01:00.068686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.068667 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" containerName="test-curl-7088-d79e1a" Oct 30 22:01:00.068686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.068685 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" containerName="test-curl-7088-d79e1a" Oct 30 22:01:00.068905 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.068885 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" containerName="test-curl-7088-d79e1a" Oct 30 22:01:00.073073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.073054 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.089152 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.089125 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.089280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.089236 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bwf\" (UniqueName: \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.124890 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.124851 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:01:00.205909 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.184848 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ea1f0b-dockercfg-9zk6x\"" Oct 30 22:01:00.205909 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.184926 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ea1f0b\"" Oct 30 22:01:00.212647 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.210359 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.212647 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.210478 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bwf\" (UniqueName: \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.212647 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.212341 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.371795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.371749 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bwf\" (UniqueName: \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\") pod \"payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.387372 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.387338 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:00.768285 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:00.768250 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:01:00.769013 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:01:00.768986 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ee4b7e_8af1_4e33_9811_a71f24cb49d5.slice/crio-f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f WatchSource:0}: Error finding container f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f: Status 404 returned error can't find the container with id f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f Oct 30 22:01:01.428927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.428891 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" event={"ID":"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5","Type":"ContainerStarted","Data":"fe7b8927218d2697c72f3f54ee6420c141094e4d20675557a597a8905819f9b6"} Oct 30 22:01:01.429323 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.428961 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" event={"ID":"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5","Type":"ContainerStarted","Data":"f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f"} Oct 30 22:01:01.430143 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.430039 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:01.550828 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.550768 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" podStartSLOduration=1.5507508570000001 podStartE2EDuration="1.550750857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:01.546415626 +0000 UTC m=+2286.220462161" watchObservedRunningTime="2026-10-30 22:01:01.550750857 +0000 UTC m=+2286.224797393" Oct 30 22:01:01.750684 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.750654 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:01:01.872400 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.872360 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\" (UniqueName: \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\") pod \"fe20ca0a-deae-4cff-b2c3-60717bafa4f0\" (UID: \"fe20ca0a-deae-4cff-b2c3-60717bafa4f0\") " Oct 30 22:01:01.874653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.874625 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct" pod "fe20ca0a-deae-4cff-b2c3-60717bafa4f0" (UID: "fe20ca0a-deae-4cff-b2c3-60717bafa4f0"). InnerVolumeSpecName "kube-api-access-6c9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:01.974257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:01.974165 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9ct\" (UniqueName: \"kubernetes.io/projected/fe20ca0a-deae-4cff-b2c3-60717bafa4f0-kube-api-access-6c9ct\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:02.436447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:02.436405 4525 generic.go:356] "Generic (PLEG): container finished" podID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" containerID="d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0" exitCode=137 Oct 30 22:01:02.436981 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:02.436463 4525 scope.go:122] "RemoveContainer" containerID="d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0" Oct 30 22:01:02.436981 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:02.436473 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:01:02.447484 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:02.447463 4525 scope.go:122] "RemoveContainer" containerID="d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0" Oct 30 22:01:02.447798 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:02.447775 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0\": container with ID starting with d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0 not found: ID does not exist" containerID="d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0" Oct 30 22:01:02.447867 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:02.447807 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0"} err="failed to get container status \"d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0\": rpc error: code = NotFound desc = could not find container \"d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0\": container with ID starting with d463b8530a3695720cb1dabd7312c49a6e13986bc9a80fe57014b981fe7bc9a0 not found: ID does not exist" Oct 30 22:01:02.496784 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:02.496749 4525 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7088-d79e1a\" is forbidden: User \"system:node:ip-10-0-134-53.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-53.us-west-1.compute.internal' and this object" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" pod="opendatahub/test-curl-7088-d79e1a" Oct 30 22:01:03.474557 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:03.474527 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe20ca0a-deae-4cff-b2c3-60717bafa4f0" path="/var/lib/kubelet/pods/fe20ca0a-deae-4cff-b2c3-60717bafa4f0/volumes" Oct 30 22:01:05.402424 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:05.402379 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:01:13.523203 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:13.519786 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:01:30.409950 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.409910 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:01:30.414638 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.414600 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.482284 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.482244 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-abb526-openshift-default-dockercfg-9mz46\"" Oct 30 22:01:30.527341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.527287 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.567886 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568034 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568088 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568118 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568156 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568184 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.573633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.568304 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.588635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.582923 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbg5\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.588635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.583083 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.732777 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.732898 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbg5\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.732939 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733007 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733126 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733178 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733207 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733245 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.734847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.733277 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.749035 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.748465 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.749638 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.749589 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.750779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.750742 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.751545 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.751521 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.751961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.751942 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.792140 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.792058 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.821529 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.817841 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.821529 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.820336 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:30.823915 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:30.823861 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbg5\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\") pod \"e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:31.029326 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.029248 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:31.471344 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:01:31.471317 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff303c6_980d_4f0a_85f7_c632f60543b6.slice/crio-c0d1e22329f1c969b421b0d6ca7006dcef4c0916e20078d6377c4b689b5abdf5 WatchSource:0}: Error finding container c0d1e22329f1c969b421b0d6ca7006dcef4c0916e20078d6377c4b689b5abdf5: Status 404 returned error can't find the container with id c0d1e22329f1c969b421b0d6ca7006dcef4c0916e20078d6377c4b689b5abdf5 Oct 30 22:01:31.474119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.474081 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:31.474225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.474179 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:31.474274 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.474233 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:31.477551 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.477531 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:01:31.797301 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.797209 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerStarted","Data":"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08"} Oct 30 22:01:31.797301 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.797280 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerStarted","Data":"c0d1e22329f1c969b421b0d6ca7006dcef4c0916e20078d6377c4b689b5abdf5"} Oct 30 22:01:31.919838 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:31.919466 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podStartSLOduration=1.919447599 podStartE2EDuration="1.919447599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:31.918111123 +0000 UTC m=+2316.592157660" watchObservedRunningTime="2026-10-30 22:01:31.919447599 +0000 UTC m=+2316.593494136" Oct 30 22:01:32.041643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:32.040659 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:32.085793 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:32.085710 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:32.804385 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:32.804359 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:32.805510 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:32.805492 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:01:35.397718 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:35.397680 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:01:39.766003 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.765968 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:01:39.770732 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.770709 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:39.874968 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.874931 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ae1bd5-serving-cert\"" Oct 30 22:01:39.879390 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.879365 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:01:39.910710 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.910673 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:39.910874 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.910745 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzk4v\" (UniqueName: \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:39.910922 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.910867 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:39.910987 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:39.910972 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.019215 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.019468 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.019513 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzk4v\" (UniqueName: \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.019565 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:40.020343 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-serving-cert: secret "maas-api-e2e-api-a-ae1bd5-serving-cert" not found Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:01:40.020479 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls podName:e03d13fc-4001-4c40-bfe7-ff6402a14ab2 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:40.52043604 +0000 UTC m=+2325.194482567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls") pod "maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" (UID: "e03d13fc-4001-4c40-bfe7-ff6402a14ab2") : secret "maas-api-e2e-api-a-ae1bd5-serving-cert" not found Oct 30 22:01:40.027907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.021848 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.029958 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.029779 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.095796 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.095762 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzk4v\" (UniqueName: \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.528864 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.528825 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.531399 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.531372 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") pod \"maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:40.686148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:40.686108 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:41.069574 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:01:41.069534 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03d13fc_4001_4c40_bfe7_ff6402a14ab2.slice/crio-1362fe1f8c25bddfdcdb5763fe677eeedaf157e6ca17f5c5009504b56747d0cc WatchSource:0}: Error finding container 1362fe1f8c25bddfdcdb5763fe677eeedaf157e6ca17f5c5009504b56747d0cc: Status 404 returned error can't find the container with id 1362fe1f8c25bddfdcdb5763fe677eeedaf157e6ca17f5c5009504b56747d0cc Oct 30 22:01:41.070670 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:41.070643 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:01:41.869244 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:41.869214 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerStarted","Data":"1362fe1f8c25bddfdcdb5763fe677eeedaf157e6ca17f5c5009504b56747d0cc"} Oct 30 22:01:42.918638 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:42.914174 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerStarted","Data":"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5"} Oct 30 22:01:42.918638 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:42.915403 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:43.035243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:43.035187 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" podStartSLOduration=3.2699954030000002 podStartE2EDuration="4.035171219s" totalImagesPullingTime="765.175816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:43.032642395 +0000 UTC m=+2327.706688949" watchObservedRunningTime="2026-10-30 22:01:43.035171219 +0000 UTC m=+2327.709217755" Oct 30 22:01:49.929029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:49.928986 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.154:8443/health\": dial tcp 10.133.0.154:8443: connect: connection refused" Oct 30 22:01:52.984812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:52.984773 4525 generic.go:356] "Generic (PLEG): container finished" podID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerID="9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5" exitCode=1 Oct 30 22:01:52.985216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:52.984848 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerDied","Data":"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5"} Oct 30 22:01:52.985367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:52.985353 4525 scope.go:122] "RemoveContainer" containerID="9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5" Oct 30 22:01:53.995303 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:53.995272 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerStarted","Data":"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48"} Oct 30 22:01:53.995733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:53.995489 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:01:59.565148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:59.565116 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 22:01:59.565548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:59.565329 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" podUID="c56444bc-4171-45a4-827d-f38176df0362" containerName="maas-api" containerID="cri-o://6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec" gracePeriod=30 Oct 30 22:01:59.926279 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:01:59.926248 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 22:02:00.035633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.019211 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\" (UniqueName: \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\") pod \"c56444bc-4171-45a4-827d-f38176df0362\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " Oct 30 22:02:00.035633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.019338 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") pod \"c56444bc-4171-45a4-827d-f38176df0362\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " Oct 30 22:02:00.035633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.019389 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\") pod \"c56444bc-4171-45a4-827d-f38176df0362\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " Oct 30 22:02:00.035633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.019460 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\") pod \"c56444bc-4171-45a4-827d-f38176df0362\" (UID: \"c56444bc-4171-45a4-827d-f38176df0362\") " Oct 30 22:02:00.035633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.020543 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca" pod "c56444bc-4171-45a4-827d-f38176df0362" (UID: "c56444bc-4171-45a4-827d-f38176df0362"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:00.052629 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.045165 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc" pod "c56444bc-4171-45a4-827d-f38176df0362" (UID: "c56444bc-4171-45a4-827d-f38176df0362"). InnerVolumeSpecName "kube-api-access-b2nhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:00.063988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.063923 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:02:00.074447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.074386 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls" pod "c56444bc-4171-45a4-827d-f38176df0362" (UID: "c56444bc-4171-45a4-827d-f38176df0362"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:00.075294 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.074716 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs" pod "c56444bc-4171-45a4-827d-f38176df0362" (UID: "c56444bc-4171-45a4-827d-f38176df0362"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:00.088871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.088831 4525 generic.go:356] "Generic (PLEG): container finished" podID="c56444bc-4171-45a4-827d-f38176df0362" containerID="6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec" exitCode=0 Oct 30 22:02:00.089137 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.089113 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" event={"ID":"c56444bc-4171-45a4-827d-f38176df0362","Type":"ContainerDied","Data":"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec"} Oct 30 22:02:00.089214 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.089175 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" event={"ID":"c56444bc-4171-45a4-827d-f38176df0362","Type":"ContainerDied","Data":"d81ad0f6779441a56380cd62cc0170037b6014ef5a07346d8ec8b3784c2cf84f"} Oct 30 22:02:00.089214 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.089207 4525 scope.go:122] "RemoveContainer" containerID="6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec" Oct 30 22:02:00.089397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.089380 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b" Oct 30 22:02:00.108920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.103440 4525 scope.go:122] "RemoveContainer" containerID="6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec" Oct 30 22:02:00.108920 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:00.103888 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec\": container with ID starting with 6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec not found: ID does not exist" containerID="6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec" Oct 30 22:02:00.108920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.103925 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec"} err="failed to get container status \"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec\": rpc error: code = NotFound desc = could not find container \"6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec\": container with ID starting with 6d1450f3aaeda558974c674dafdef245340f267f826e147629e7b401d54338ec not found: ID does not exist" Oct 30 22:02:00.122547 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.122518 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2nhc\" (UniqueName: \"kubernetes.io/projected/c56444bc-4171-45a4-827d-f38176df0362-kube-api-access-b2nhc\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.122547 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.122550 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.122773 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.122565 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56444bc-4171-45a4-827d-f38176df0362-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.122773 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.122579 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56444bc-4171-45a4-827d-f38176df0362-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:00.320523 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.320431 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 22:02:00.329182 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:00.329142 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71a3d788-56cf844b5f-p9g6b"] Oct 30 22:02:01.474769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:01.474737 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56444bc-4171-45a4-827d-f38176df0362" path="/var/lib/kubelet/pods/c56444bc-4171-45a4-827d-f38176df0362/volumes" Oct 30 22:02:02.264883 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:02.264350 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:02:05.418100 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:05.418012 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:02:11.931764 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:11.931723 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:02:11.932164 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:11.931802 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:13.791695 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:13.791660 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 22:02:14.284728 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.284698 4525 generic.go:356] "Generic (PLEG): container finished" podID="4b544a7d-7df3-4799-841c-f5069200e931" containerID="36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab" exitCode=0 Oct 30 22:02:14.284940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.284770 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerDied","Data":"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab"} Oct 30 22:02:14.285280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.285259 4525 scope.go:122] "RemoveContainer" containerID="36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab" Oct 30 22:02:14.286794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.286774 4525 generic.go:356] "Generic (PLEG): container finished" podID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerID="6b33408d899402aaa07623e8f87042ae60ff056cdb5c3c407862efd27bf47849" exitCode=0 Oct 30 22:02:14.286895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.286852 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerDied","Data":"6b33408d899402aaa07623e8f87042ae60ff056cdb5c3c407862efd27bf47849"} Oct 30 22:02:14.287023 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.286995 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.287077 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.287064 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.287114 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.287101 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.287206 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.287189 4525 scope.go:122] "RemoveContainer" containerID="6b33408d899402aaa07623e8f87042ae60ff056cdb5c3c407862efd27bf47849" Oct 30 22:02:14.288976 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.288950 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.289053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.289024 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.289087 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.289068 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:14.970279 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:14.970245 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:02:15.295457 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:15.295362 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerStarted","Data":"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027"} Oct 30 22:02:15.297408 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:15.297384 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerStarted","Data":"20e385ec3fffdcceffb3ba4942c4c9b5996acfa5ae03752cff09a647ef68a04f"} Oct 30 22:02:15.626339 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:15.626310 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:02:15.626540 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:15.626520 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.626587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:15.626570 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:16.182805 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:16.182767 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:02:16.184126 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:16.184101 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:02:16.184334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:16.184303 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:02:16.626572 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:16.626535 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:16.626744 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:16.626600 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:17.182984 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:17.182952 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:02:17.183517 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:17.183479 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:02:17.627086 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:17.627045 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:17.627279 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:17.627108 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:17.717835 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:17.717801 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 22:02:18.028442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.028356 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:02:18.029119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.029103 4525 cpu_manager.go:414] "containerMap: removing container" podUID="c56444bc-4171-45a4-827d-f38176df0362" containerName="maas-api" Oct 30 22:02:18.029187 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.029121 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56444bc-4171-45a4-827d-f38176df0362" containerName="maas-api" Oct 30 22:02:18.029244 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.029235 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56444bc-4171-45a4-827d-f38176df0362" containerName="maas-api" Oct 30 22:02:18.067115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.067081 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.092867 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.092829 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:02:18.120440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.120409 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ea1f0b-serving-cert\"" Oct 30 22:02:18.122117 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.122086 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.122258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.122215 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.122355 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.122341 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.122408 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.122384 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptt6g\" (UniqueName: \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.188286 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.188251 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:02:18.188918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.188885 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:02:18.223828 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.223788 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.224020 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.223989 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.224085 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.224059 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptt6g\" (UniqueName: \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.224158 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.224141 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.224356 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:18.224336 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-serving-cert: secret "maas-api-e2e-inf-b-ea1f0b-serving-cert" not found Oct 30 22:02:18.224418 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:18.224408 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls podName:a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e nodeName:}" failed. No retries permitted until 2026-10-30 22:02:18.724388148 +0000 UTC m=+2363.398434663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls") pod "maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" (UID: "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e") : secret "maas-api-e2e-inf-b-ea1f0b-serving-cert" not found Oct 30 22:02:18.225574 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.225547 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.228435 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.228411 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.310105 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.309975 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptt6g\" (UniqueName: \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.626599 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.626560 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:18.626795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.626625 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:18.777632 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.777529 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:18.784633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:18.784564 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:19.001377 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.001289 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:19.194663 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.189048 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:02:19.194663 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.189121 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:02:19.491831 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.491799 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:02:19.626998 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.626964 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:19.627179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:19.627036 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:20.183055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.183015 4525 patch_prober.go:29] interesting pod/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:02:20.183294 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.183083 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 22:02:20.292631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.277039 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 22:02:20.392305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.392108 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 22:02:20.401559 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.401529 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" event={"ID":"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e","Type":"ContainerStarted","Data":"0679902e41a9a124a4aa29a9f34362ab330a6729836b8a459c9937f52399e12c"} Oct 30 22:02:20.626862 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.626831 4525 patch_prober.go:29] interesting pod/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:02:20.627032 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:20.626877 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 22:02:21.188464 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.188437 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:02:21.188881 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.188860 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:02:21.189689 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.189669 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:02:21.420560 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.420507 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" event={"ID":"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e","Type":"ContainerStarted","Data":"2676062877e0cac05b580e2fee9df68fd505c090d1b5af1ef1f434f61cc0c078"} Oct 30 22:02:21.421014 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.420748 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:21.589988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.589879 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" podStartSLOduration=3.763988756 podStartE2EDuration="4.589862138s" totalImagesPullingTime="825.873382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:21.587263461 +0000 UTC m=+2366.261310025" watchObservedRunningTime="2026-10-30 22:02:21.589862138 +0000 UTC m=+2366.263908730" Oct 30 22:02:21.630707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.630680 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:02:21.631170 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.631144 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:02:21.631854 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:21.631835 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:02:25.294656 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:25.294568 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" containerID="cri-o://c7bbc84ac2b9cb435fe800f4752c51f3954cc87c7cb84d1525cce7e932c936a6" gracePeriod=25 Oct 30 22:02:25.392434 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:25.392386 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" containerID="cri-o://2d7539201cef2539761284be81967b1fa4908b24d9dbd3a53d28e860a88cd4b7" gracePeriod=25 Oct 30 22:02:25.669440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:25.669399 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:27.433621 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:27.433576 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:02:28.814722 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:28.814673 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" containerID="cri-o://2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934" gracePeriod=45 Oct 30 22:02:29.087071 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.087045 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 22:02:29.158495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158456 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158547 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158573 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158657 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158683 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158818 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\" (UniqueName: \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\") pod \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\" (UID: \"ed5bd236-ca76-47db-8d20-64df7cdd4e5e\") " Oct 30 22:02:29.158983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158897 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.158983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.158937 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.159135 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.159011 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.159401 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.159377 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.159401 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.159401 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.159771 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.159412 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.161042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.161014 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "kube-api-access-5pld2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:29.161139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.161036 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:29.161139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.161061 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm" pod "ed5bd236-ca76-47db-8d20-64df7cdd4e5e" (UID: "ed5bd236-ca76-47db-8d20-64df7cdd4e5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:29.261781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.261743 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pld2\" (UniqueName: \"kubernetes.io/projected/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-kube-api-access-5pld2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.261781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.261782 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.262034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.261802 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5bd236-ca76-47db-8d20-64df7cdd4e5e-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:29.491198 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.491167 4525 generic.go:356] "Generic (PLEG): container finished" podID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerID="2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934" exitCode=0 Oct 30 22:02:29.491316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.491215 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" Oct 30 22:02:29.491316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.491241 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" event={"ID":"ed5bd236-ca76-47db-8d20-64df7cdd4e5e","Type":"ContainerDied","Data":"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934"} Oct 30 22:02:29.491316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.491283 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk" event={"ID":"ed5bd236-ca76-47db-8d20-64df7cdd4e5e","Type":"ContainerDied","Data":"9ba4b18e445631f5d685468039e7850e83286e556b01d5f71cd803c944663950"} Oct 30 22:02:29.491453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.491337 4525 scope.go:122] "RemoveContainer" containerID="2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934" Oct 30 22:02:29.502039 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.502021 4525 scope.go:122] "RemoveContainer" containerID="2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934" Oct 30 22:02:29.502337 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:29.502310 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934\": container with ID starting with 2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934 not found: ID does not exist" containerID="2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934" Oct 30 22:02:29.502434 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.502342 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934"} err="failed to get container status \"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934\": rpc error: code = NotFound desc = could not find container \"2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934\": container with ID starting with 2b9af9a93cdfe5ee249d66f140af52686002fd5e2f1958b8f2e1c127db3b3934 not found: ID does not exist" Oct 30 22:02:29.660162 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.660128 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 22:02:29.710007 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:29.709968 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-91874e/premium-w2-91874e-kserve-575bcd577b-tfhsk"] Oct 30 22:02:31.474379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:31.474339 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" path="/var/lib/kubelet/pods/ed5bd236-ca76-47db-8d20-64df7cdd4e5e/volumes" Oct 30 22:02:32.802282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:32.802247 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerName="main" containerID="cri-o://33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519" gracePeriod=45 Oct 30 22:02:33.086932 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.086905 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222297 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222500 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\" (UniqueName: \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222539 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222658 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222697 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.222848 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\") pod \"6de23ea9-9274-4d0c-932e-6858f37d03cf\" (UID: \"6de23ea9-9274-4d0c-932e-6858f37d03cf\") " Oct 30 22:02:33.224730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.224474 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:33.225736 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.225553 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:33.226359 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.226059 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:33.233358 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.233319 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:33.234666 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.234568 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:33.237757 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.237717 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq" pod "6de23ea9-9274-4d0c-932e-6858f37d03cf" (UID: "6de23ea9-9274-4d0c-932e-6858f37d03cf"). InnerVolumeSpecName "kube-api-access-tgnnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360080 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de23ea9-9274-4d0c-932e-6858f37d03cf-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360109 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360121 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgnnq\" (UniqueName: \"kubernetes.io/projected/6de23ea9-9274-4d0c-932e-6858f37d03cf-kube-api-access-tgnnq\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360131 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360141 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.360225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.360149 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de23ea9-9274-4d0c-932e-6858f37d03cf-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:33.520032 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.519998 4525 generic.go:356] "Generic (PLEG): container finished" podID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerID="33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519" exitCode=0 Oct 30 22:02:33.520220 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.520061 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" Oct 30 22:02:33.520220 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.520079 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" event={"ID":"6de23ea9-9274-4d0c-932e-6858f37d03cf","Type":"ContainerDied","Data":"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519"} Oct 30 22:02:33.520220 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.520122 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" event={"ID":"6de23ea9-9274-4d0c-932e-6858f37d03cf","Type":"ContainerDied","Data":"4e902f08c94a3f592c9d38829154bd49767a8b63e41cc93a20fcd59a05361b56"} Oct 30 22:02:33.520220 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.520140 4525 scope.go:122] "RemoveContainer" containerID="33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519" Oct 30 22:02:33.530517 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.530493 4525 scope.go:122] "RemoveContainer" containerID="33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519" Oct 30 22:02:33.530874 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:33.530851 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519\": container with ID starting with 33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519 not found: ID does not exist" containerID="33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519" Oct 30 22:02:33.530937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.530886 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519"} err="failed to get container status \"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519\": rpc error: code = NotFound desc = could not find container \"33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519\": container with ID starting with 33593e63551409a6a8a35b7570c745fb431fd4e66989010d524744a5bd5f2519 not found: ID does not exist" Oct 30 22:02:33.687894 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.687858 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 22:02:33.737561 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:33.737523 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9"] Oct 30 22:02:34.075629 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:34.075487 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-71a3d788/e2e-lifecycle-model-71a3d788-kserve-6887f99565-mjhz9" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:34.663881 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:34.663844 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:35.397881 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:02:35.397844 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:02:35.476813 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:35.476784 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" path="/var/lib/kubelet/pods/6de23ea9-9274-4d0c-932e-6858f37d03cf/volumes" Oct 30 22:02:35.669467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:35.669386 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:44.663477 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:44.663390 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:02:44.759627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:44.759566 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:02:45.639430 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.639396 4525 generic.go:356] "Generic (PLEG): container finished" podID="cead6166-5968-4180-bad3-6cc47ceec634" containerID="c7bbc84ac2b9cb435fe800f4752c51f3954cc87c7cb84d1525cce7e932c936a6" exitCode=0 Oct 30 22:02:45.639726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.639707 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" event={"ID":"cead6166-5968-4180-bad3-6cc47ceec634","Type":"ContainerDied","Data":"c7bbc84ac2b9cb435fe800f4752c51f3954cc87c7cb84d1525cce7e932c936a6"} Oct 30 22:02:45.641975 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.641950 4525 generic.go:356] "Generic (PLEG): container finished" podID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerID="2d7539201cef2539761284be81967b1fa4908b24d9dbd3a53d28e860a88cd4b7" exitCode=0 Oct 30 22:02:45.642403 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.642139 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" event={"ID":"bdf53135-9a0f-4c4c-aa43-8c4452773a20","Type":"ContainerDied","Data":"2d7539201cef2539761284be81967b1fa4908b24d9dbd3a53d28e860a88cd4b7"} Oct 30 22:02:45.804248 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.804216 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 22:02:45.922957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.905324 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\") pod \"cead6166-5968-4180-bad3-6cc47ceec634\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " Oct 30 22:02:45.922957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.905413 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\" (UniqueName: \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\") pod \"cead6166-5968-4180-bad3-6cc47ceec634\" (UID: \"cead6166-5968-4180-bad3-6cc47ceec634\") " Oct 30 22:02:45.922957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.921598 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume" pod "cead6166-5968-4180-bad3-6cc47ceec634" (UID: "cead6166-5968-4180-bad3-6cc47ceec634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:45.924278 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.924250 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cead6166-5968-4180-bad3-6cc47ceec634-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:45.935794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.935762 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl" pod "cead6166-5968-4180-bad3-6cc47ceec634" (UID: "cead6166-5968-4180-bad3-6cc47ceec634"). InnerVolumeSpecName "kube-api-access-clfcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:45.959042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:45.959015 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 22:02:46.025435 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.025406 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfcl\" (UniqueName: \"kubernetes.io/projected/cead6166-5968-4180-bad3-6cc47ceec634-kube-api-access-clfcl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.126090 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.126056 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\") pod \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " Oct 30 22:02:46.126281 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.126211 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\" (UniqueName: \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\") pod \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\" (UID: \"bdf53135-9a0f-4c4c-aa43-8c4452773a20\") " Oct 30 22:02:46.126434 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.126410 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume" pod "bdf53135-9a0f-4c4c-aa43-8c4452773a20" (UID: "bdf53135-9a0f-4c4c-aa43-8c4452773a20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:46.126890 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.126869 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf53135-9a0f-4c4c-aa43-8c4452773a20-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.128561 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.128538 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f" pod "bdf53135-9a0f-4c4c-aa43-8c4452773a20" (UID: "bdf53135-9a0f-4c4c-aa43-8c4452773a20"). InnerVolumeSpecName "kube-api-access-n9v9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:46.228125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.228050 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9v9f\" (UniqueName: \"kubernetes.io/projected/bdf53135-9a0f-4c4c-aa43-8c4452773a20-kube-api-access-n9v9f\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.663007 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.662973 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 22:02:46.663196 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.662979 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" event={"ID":"cead6166-5968-4180-bad3-6cc47ceec634","Type":"ContainerDied","Data":"3a054c12402b63ab0f2d899d04a3745b3d081d7b4b195eb5357095fcfd248917"} Oct 30 22:02:46.663196 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.663115 4525 scope.go:122] "RemoveContainer" containerID="c7bbc84ac2b9cb435fe800f4752c51f3954cc87c7cb84d1525cce7e932c936a6" Oct 30 22:02:46.664745 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.664721 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" event={"ID":"bdf53135-9a0f-4c4c-aa43-8c4452773a20","Type":"ContainerDied","Data":"e249bba9569403ce0893e5a3b1611ff401e98b10a474b2e0465a82c86ce40cff"} Oct 30 22:02:46.664862 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.664811 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99" Oct 30 22:02:46.669135 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.668965 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded" Oct 30 22:02:46.669135 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:02:46.669065 4525 logging.go:55] [core] [Channel #4649 SubChannel #4650]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.47:50052", ServerName: "10.133.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.47:50052: operation was canceled" Oct 30 22:02:46.669135 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.669086 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227" Oct 30 22:02:46.674452 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.674434 4525 scope.go:122] "RemoveContainer" containerID="2d7539201cef2539761284be81967b1fa4908b24d9dbd3a53d28e860a88cd4b7" Oct 30 22:02:46.856888 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.856849 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 22:02:46.862248 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.862211 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-91874e-5fb698f8ff-nkd99"] Oct 30 22:02:46.961947 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.961853 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 22:02:46.966137 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:46.966111 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-91874e-797b6cf85d-cc227"] Oct 30 22:02:47.513633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:47.513333 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" path="/var/lib/kubelet/pods/bdf53135-9a0f-4c4c-aa43-8c4452773a20/volumes" Oct 30 22:02:47.517640 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:47.514186 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cead6166-5968-4180-bad3-6cc47ceec634" path="/var/lib/kubelet/pods/cead6166-5968-4180-bad3-6cc47ceec634/volumes" Oct 30 22:02:56.743825 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.743786 4525 generic.go:356] "Generic (PLEG): container finished" podID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerID="7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87" exitCode=0 Oct 30 22:02:56.744333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.743863 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerDied","Data":"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87"} Oct 30 22:02:56.744474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.744458 4525 scope.go:122] "RemoveContainer" containerID="7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87" Oct 30 22:02:56.746190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.746160 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:56.746246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.746234 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:56.746282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:56.746268 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:57.764340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:57.764301 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerStarted","Data":"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf"} Oct 30 22:02:58.101522 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:58.101426 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:02:58.102927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:58.102899 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:02:58.103064 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:58.102959 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:02:59.101451 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.101416 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:02:59.101945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.101478 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:02:59.317639 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.311731 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:02:59.777936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.777904 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:02:59.778554 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778541 4525 cpu_manager.go:414] "containerMap: removing container" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" Oct 30 22:02:59.778622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778555 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" Oct 30 22:02:59.778622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778570 4525 cpu_manager.go:414] "containerMap: removing container" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" Oct 30 22:02:59.778622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778575 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" Oct 30 22:02:59.778725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778630 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerName="main" Oct 30 22:02:59.778725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778638 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerName="main" Oct 30 22:02:59.778725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778667 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" Oct 30 22:02:59.778725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778672 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" Oct 30 22:02:59.778861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778779 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de23ea9-9274-4d0c-932e-6858f37d03cf" containerName="main" Oct 30 22:02:59.778861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778791 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5bd236-ca76-47db-8d20-64df7cdd4e5e" containerName="main" Oct 30 22:02:59.778861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778799 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cead6166-5968-4180-bad3-6cc47ceec634" containerName="payload-processing" Oct 30 22:02:59.778861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.778806 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf53135-9a0f-4c4c-aa43-8c4452773a20" containerName="payload-pre-processing" Oct 30 22:02:59.781311 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.781294 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:02:59.855634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.852821 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19bc92\"" Oct 30 22:02:59.901633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.901230 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19bc92-dockercfg-gcsh5\"" Oct 30 22:02:59.934631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.934526 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:02:59.951643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.934954 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzpj\" (UniqueName: \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:02:59.966819 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:02:59.966032 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:03:00.080568 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.080478 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzpj\" (UniqueName: \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:00.080568 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.080565 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:00.081645 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.081572 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:00.092005 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.091967 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:03:00.096656 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.096625 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.098088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.098058 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:03:00.101428 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.101402 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:03:00.101584 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.101460 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:03:00.160696 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.160631 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzpj\" (UniqueName: \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\") pod \"payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:00.282874 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.282835 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.283103 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.283084 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwzm\" (UniqueName: \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.384199 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.384157 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwzm\" (UniqueName: \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.384394 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.384244 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.385006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.384983 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.395219 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.395193 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:00.461060 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.461021 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwzm\" (UniqueName: \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\") pod \"payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.715654 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.715543 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:00.822833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:00.821997 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:03:01.102067 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.101975 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:03:01.102067 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.102033 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:03:01.123556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.123521 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:03:01.124974 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:03:01.124946 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d205b6_099b_47c6_a8b3_c6e18380473f.slice/crio-9a7aaefaf9bedb713414906ae1d16b211ff909bab73be4f3a8e73e60bddd717c WatchSource:0}: Error finding container 9a7aaefaf9bedb713414906ae1d16b211ff909bab73be4f3a8e73e60bddd717c: Status 404 returned error can't find the container with id 9a7aaefaf9bedb713414906ae1d16b211ff909bab73be4f3a8e73e60bddd717c Oct 30 22:03:01.794063 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.794028 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" event={"ID":"46d205b6-099b-47c6-a8b3-c6e18380473f","Type":"ContainerStarted","Data":"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97"} Oct 30 22:03:01.794264 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.794092 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" event={"ID":"46d205b6-099b-47c6-a8b3-c6e18380473f","Type":"ContainerStarted","Data":"9a7aaefaf9bedb713414906ae1d16b211ff909bab73be4f3a8e73e60bddd717c"} Oct 30 22:03:01.794316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.794263 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:01.795917 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.795893 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" event={"ID":"1672e201-f9de-436b-a775-f0617c9dbfbd","Type":"ContainerStarted","Data":"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5"} Oct 30 22:03:01.795917 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.795931 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" event={"ID":"1672e201-f9de-436b-a775-f0617c9dbfbd","Type":"ContainerStarted","Data":"83614783da3f5a09f03fbbfbb340d577b3afad82a71a1b7a7c9614222b99aa80"} Oct 30 22:03:01.796139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.796045 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:01.915719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:01.915659 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" podStartSLOduration=2.915645981 podStartE2EDuration="2.915645981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:01.914015988 +0000 UTC m=+2406.588062526" watchObservedRunningTime="2026-10-30 22:03:01.915645981 +0000 UTC m=+2406.589692553" Oct 30 22:03:02.036632 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:02.032291 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" podStartSLOduration=3.032270675 podStartE2EDuration="3.032270675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:02.030950454 +0000 UTC m=+2406.704997001" watchObservedRunningTime="2026-10-30 22:03:02.032270675 +0000 UTC m=+2406.706317212" Oct 30 22:03:02.102682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:02.101742 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:03:02.102682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:02.101823 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:03:03.102112 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:03.102073 4525 patch_prober.go:29] interesting pod/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 22:03:03.102330 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:03.102133 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 22:03:04.108785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:04.108733 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:03:04.111317 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:04.111292 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:03:05.459005 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:03:05.458972 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:03:12.782629 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.782077 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:03:12.794740 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.792976 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:12.837702 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.836384 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:03:12.854892 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.848741 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:03:12.854892 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.852941 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:03:12.904190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.904158 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5bf4f8\"" Oct 30 22:03:12.916830 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.905983 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5bf4f8-dockercfg-pfvnh\"" Oct 30 22:03:12.922685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.907084 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:12.922685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.921217 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6779\" (UniqueName: \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:12.958519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.958484 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:03:12.962351 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:12.962328 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.002308 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.002275 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:03:13.022377 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.022336 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.022555 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.022510 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6779\" (UniqueName: \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.023179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.023160 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.085360 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.085284 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6779\" (UniqueName: \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\") pod \"payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.124458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.124418 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzc7\" (UniqueName: \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.124738 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.124558 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.160371 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.160338 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.225926 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.225538 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzc7\" (UniqueName: \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.225926 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.225656 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.226648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.226601 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.290372 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.290332 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzc7\" (UniqueName: \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\") pod \"payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.562845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.562815 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:03:13.563178 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:03:13.563150 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafe5ae7_f024_4175_8152_d870fec17fae.slice/crio-ec5c1105ecb5a0baa10e8e254f4bb7643837ff82679644136ba18db9ac7dbbde WatchSource:0}: Error finding container ec5c1105ecb5a0baa10e8e254f4bb7643837ff82679644136ba18db9ac7dbbde: Status 404 returned error can't find the container with id ec5c1105ecb5a0baa10e8e254f4bb7643837ff82679644136ba18db9ac7dbbde Oct 30 22:03:13.575489 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.575458 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:13.924037 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.924004 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" event={"ID":"bafe5ae7-f024-4175-8152-d870fec17fae","Type":"ContainerStarted","Data":"752e3f9615b95862ecd0e37296b2ab80cc57312fa2a103538a3e1b176751c64e"} Oct 30 22:03:13.924462 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.924065 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" event={"ID":"bafe5ae7-f024-4175-8152-d870fec17fae","Type":"ContainerStarted","Data":"ec5c1105ecb5a0baa10e8e254f4bb7643837ff82679644136ba18db9ac7dbbde"} Oct 30 22:03:13.924462 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.924094 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:13.961743 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:13.961717 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:03:13.963587 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:03:13.963559 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6d903d_ad05_4df9_ab60_7714a6c5583a.slice/crio-1658ab3053127fb71823046c5355b4745ffd58c2af0f014ea71d04ffcc125089 WatchSource:0}: Error finding container 1658ab3053127fb71823046c5355b4745ffd58c2af0f014ea71d04ffcc125089: Status 404 returned error can't find the container with id 1658ab3053127fb71823046c5355b4745ffd58c2af0f014ea71d04ffcc125089 Oct 30 22:03:14.045277 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:14.045227 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" podStartSLOduration=2.045208903 podStartE2EDuration="2.045208903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:14.044975732 +0000 UTC m=+2418.719022269" watchObservedRunningTime="2026-10-30 22:03:14.045208903 +0000 UTC m=+2418.719255440" Oct 30 22:03:14.933269 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:14.933228 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" event={"ID":"ec6d903d-ad05-4df9-ab60-7714a6c5583a","Type":"ContainerStarted","Data":"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2"} Oct 30 22:03:14.933759 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:14.933295 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" event={"ID":"ec6d903d-ad05-4df9-ab60-7714a6c5583a","Type":"ContainerStarted","Data":"1658ab3053127fb71823046c5355b4745ffd58c2af0f014ea71d04ffcc125089"} Oct 30 22:03:15.052535 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:15.052472 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" podStartSLOduration=3.052458804 podStartE2EDuration="3.052458804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.051941902 +0000 UTC m=+2419.725988440" watchObservedRunningTime="2026-10-30 22:03:15.052458804 +0000 UTC m=+2419.726505340" Oct 30 22:03:15.939885 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:15.939854 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:20.119704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.119667 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:03:20.123291 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.123273 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.235686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.228028 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5a19c1\"" Oct 30 22:03:20.235686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.235140 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:03:20.235686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.235297 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5a19c1-dockercfg-b8vxp\"" Oct 30 22:03:20.319690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.288484 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvwg\" (UniqueName: \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.320316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.320098 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.418406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.417721 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:03:20.422687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.422591 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvwg\" (UniqueName: \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.422883 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.422763 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.423835 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.423814 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.424258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.423881 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.427945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.427923 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:03:20.485528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.485488 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvwg\" (UniqueName: \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\") pod \"payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.524983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.524943 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d797p\" (UniqueName: \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.525191 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.524998 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.626994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.626960 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d797p\" (UniqueName: \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.627174 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.627013 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.627819 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.627798 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.689763 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.689682 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d797p\" (UniqueName: \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\") pod \"payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:20.737113 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.737079 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:20.748116 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:20.748090 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:21.125257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.125230 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:03:21.125681 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:03:21.125659 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8034a8ec_b2fd_4d29_8d8b_71b2bb59ba06.slice/crio-8e33ef6dc9aecae772871175cd85a1eae341c34b41c7aac24c3efd3716b63510 WatchSource:0}: Error finding container 8e33ef6dc9aecae772871175cd85a1eae341c34b41c7aac24c3efd3716b63510: Status 404 returned error can't find the container with id 8e33ef6dc9aecae772871175cd85a1eae341c34b41c7aac24c3efd3716b63510 Oct 30 22:03:21.175777 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.175748 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:03:21.177080 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:03:21.177050 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839bddb2_d550_48b3_8043_a2beac8bd4b2.slice/crio-a0271d2c51c533bd401836543236be8bdf1e518f322efaa729505128a7e3ef07 WatchSource:0}: Error finding container a0271d2c51c533bd401836543236be8bdf1e518f322efaa729505128a7e3ef07: Status 404 returned error can't find the container with id a0271d2c51c533bd401836543236be8bdf1e518f322efaa729505128a7e3ef07 Oct 30 22:03:21.983085 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.983047 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" event={"ID":"839bddb2-d550-48b3-8043-a2beac8bd4b2","Type":"ContainerStarted","Data":"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9"} Oct 30 22:03:21.983333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.983109 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:21.983333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.983131 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" event={"ID":"839bddb2-d550-48b3-8043-a2beac8bd4b2","Type":"ContainerStarted","Data":"a0271d2c51c533bd401836543236be8bdf1e518f322efaa729505128a7e3ef07"} Oct 30 22:03:21.984909 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.984883 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" event={"ID":"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06","Type":"ContainerStarted","Data":"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f"} Oct 30 22:03:21.985053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.984925 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" event={"ID":"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06","Type":"ContainerStarted","Data":"8e33ef6dc9aecae772871175cd85a1eae341c34b41c7aac24c3efd3716b63510"} Oct 30 22:03:21.985053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:21.985022 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:22.100656 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:22.100574 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" podStartSLOduration=2.100558326 podStartE2EDuration="2.100558326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:22.098978936 +0000 UTC m=+2426.773025473" watchObservedRunningTime="2026-10-30 22:03:22.100558326 +0000 UTC m=+2426.774604861" Oct 30 22:03:24.937030 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:24.936999 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:03:25.058199 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:25.058142 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" podStartSLOduration=5.058126507 podStartE2EDuration="5.058126507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:22.221859197 +0000 UTC m=+2426.895905765" watchObservedRunningTime="2026-10-30 22:03:25.058126507 +0000 UTC m=+2429.732173042" Oct 30 22:03:26.951285 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:26.951258 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:03:32.994987 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:32.994957 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:03:32.995447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:32.995035 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:03:35.396842 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:03:35.396795 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:03:43.116970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.116929 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 22:03:43.117497 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.117194 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" containerID="cri-o://816ab21c5773e6064ee0953ca8ec37b81494f4dd51ec7d0aeb56d990ed7b9740" gracePeriod=30 Oct 30 22:03:43.133309 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.133276 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:03:43.133500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.133480 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" containerID="cri-o://8ceba82ef7820c416340101889b48b5aae455baeb1e734a4d6264f0dc0a4a8e6" gracePeriod=30 Oct 30 22:03:43.159057 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.159020 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:03:43.159321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.159281 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" containerID="cri-o://56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa" gracePeriod=30 Oct 30 22:03:43.609245 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.609163 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:43.609245 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:43.609232 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:46.569902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:46.569862 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:46.570314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:46.569923 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:46.898349 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:46.898297 4525 patch_prober.go:29] interesting pod/e2e-mt-71a3d788-openshift-default-778558588f-snq52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:46.898550 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:46.898361 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:48.268055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.268020 4525 generic.go:356] "Generic (PLEG): container finished" podID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerID="816ab21c5773e6064ee0953ca8ec37b81494f4dd51ec7d0aeb56d990ed7b9740" exitCode=0 Oct 30 22:03:48.268443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.268091 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerDied","Data":"816ab21c5773e6064ee0953ca8ec37b81494f4dd51ec7d0aeb56d990ed7b9740"} Oct 30 22:03:48.268443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.268176 4525 scope.go:122] "RemoveContainer" containerID="8b8abf001bcacf4c0a475268b3abd4cca1e04714fd599b9bf077065c60d6b573" Oct 30 22:03:48.270473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.270452 4525 generic.go:356] "Generic (PLEG): container finished" podID="645f9f90-95b2-42e9-9808-c0db83424296" containerID="8ceba82ef7820c416340101889b48b5aae455baeb1e734a4d6264f0dc0a4a8e6" exitCode=0 Oct 30 22:03:48.270598 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.270487 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" event={"ID":"645f9f90-95b2-42e9-9808-c0db83424296","Type":"ContainerDied","Data":"8ceba82ef7820c416340101889b48b5aae455baeb1e734a4d6264f0dc0a4a8e6"} Oct 30 22:03:48.298137 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.298089 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:03:48.608246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.608215 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:03:48.636053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.636025 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:03:48.658664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.658640 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 22:03:48.684880 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684852 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684891 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684913 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684938 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684955 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.684982 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.685025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685016 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685046 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685079 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685117 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685179 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685192 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685204 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685282 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685360 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685416 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685448 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685478 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685185 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685514 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685397 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685495 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685501 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685651 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.685761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.685666 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686033 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686068 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686117 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686186 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686231 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.686363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686272 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.686712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686501 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.686712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686544 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.686712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686590 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.686712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686667 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.686712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686694 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.686989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686732 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\") pod \"645f9f90-95b2-42e9-9808-c0db83424296\" (UID: \"645f9f90-95b2-42e9-9808-c0db83424296\") " Oct 30 22:03:48.686989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686766 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\") pod \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\" (UID: \"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4\") " Oct 30 22:03:48.686989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686800 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.686989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.686834 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\") pod \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\" (UID: \"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce\") " Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687528 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687565 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687580 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687594 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687625 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645f9f90-95b2-42e9-9808-c0db83424296-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687639 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687652 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687666 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687680 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687694 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687706 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687721 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687734 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.687910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.687767 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.688949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.688040 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "kube-api-access-ww8hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.688949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.688199 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.688949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.688308 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.689301 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.689265 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:48.689996 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.689970 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "kube-api-access-sndt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.690139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690103 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.690139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690107 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.690267 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690210 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.690533 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690507 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo" pod "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" (UID: "c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.690660 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690639 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "kube-api-access-gfhxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.690933 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.690912 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:48.691055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.691042 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo" pod "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" (UID: "99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:48.691531 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.691516 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy" pod "645f9f90-95b2-42e9-9808-c0db83424296" (UID: "645f9f90-95b2-42e9-9808-c0db83424296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789524 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sndt7\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-kube-api-access-sndt7\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789557 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789570 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789584 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645f9f90-95b2-42e9-9808-c0db83424296-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789593 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789618 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789632 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645f9f90-95b2-42e9-9808-c0db83424296-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789642 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhxl\" (UniqueName: \"kubernetes.io/projected/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-kube-api-access-gfhxl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789650 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789661 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789669 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww8hl\" (UniqueName: \"kubernetes.io/projected/645f9f90-95b2-42e9-9808-c0db83424296-kube-api-access-ww8hl\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789679 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789689 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.789978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:48.789702 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:49.278850 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.278818 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" Oct 30 22:03:49.279311 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.278849 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52" event={"ID":"645f9f90-95b2-42e9-9808-c0db83424296","Type":"ContainerDied","Data":"4f780a5b10f24cf097d0be1a1cff1d7069093b592944e508ed3c62e6da0a10cf"} Oct 30 22:03:49.279311 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.278907 4525 scope.go:122] "RemoveContainer" containerID="8ceba82ef7820c416340101889b48b5aae455baeb1e734a4d6264f0dc0a4a8e6" Oct 30 22:03:49.282043 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.282025 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" Oct 30 22:03:49.282177 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.282043 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs" event={"ID":"c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce","Type":"ContainerDied","Data":"ac53302410d63225daa7f94371a672eba24adfa82885381fb99d27ded958a6fb"} Oct 30 22:03:49.283556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.283530 4525 generic.go:356] "Generic (PLEG): container finished" podID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerID="56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa" exitCode=0 Oct 30 22:03:49.283733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.283579 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" Oct 30 22:03:49.283733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.283639 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" event={"ID":"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4","Type":"ContainerDied","Data":"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa"} Oct 30 22:03:49.283733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.283690 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj" event={"ID":"99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4","Type":"ContainerDied","Data":"9329766eaa46278f069d00686775178db13d5226c245cc713f7a7b27007ede39"} Oct 30 22:03:49.291241 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.291218 4525 scope.go:122] "RemoveContainer" containerID="816ab21c5773e6064ee0953ca8ec37b81494f4dd51ec7d0aeb56d990ed7b9740" Oct 30 22:03:49.301051 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.301035 4525 scope.go:122] "RemoveContainer" containerID="56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa" Oct 30 22:03:49.311623 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.311592 4525 scope.go:122] "RemoveContainer" containerID="56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa" Oct 30 22:03:49.311873 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:03:49.311855 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa\": container with ID starting with 56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa not found: ID does not exist" containerID="56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa" Oct 30 22:03:49.311930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.311880 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa"} err="failed to get container status \"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa\": rpc error: code = NotFound desc = could not find container \"56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa\": container with ID starting with 56c134f32a6fd09a154840ceb22827881faa27f63aeaf1194944deae4e38caaa not found: ID does not exist" Oct 30 22:03:49.459258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.459157 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:03:49.463650 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.463622 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-snq52"] Oct 30 22:03:49.474857 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.474827 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645f9f90-95b2-42e9-9808-c0db83424296" path="/var/lib/kubelet/pods/645f9f90-95b2-42e9-9808-c0db83424296/volumes" Oct 30 22:03:49.584722 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.584684 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 22:03:49.660468 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.660412 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-wfkvs"] Oct 30 22:03:49.764169 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.764087 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:03:49.772994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:49.772965 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71a3d788-openshift-default-778558588f-c8mwj"] Oct 30 22:03:51.475262 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:51.475222 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" path="/var/lib/kubelet/pods/99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4/volumes" Oct 30 22:03:51.475756 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:51.475741 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" path="/var/lib/kubelet/pods/c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce/volumes" Oct 30 22:03:59.362717 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.362684 4525 generic.go:356] "Generic (PLEG): container finished" podID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerID="fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08" exitCode=0 Oct 30 22:03:59.363166 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.362737 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerDied","Data":"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08"} Oct 30 22:03:59.363243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.363228 4525 scope.go:122] "RemoveContainer" containerID="fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08" Oct 30 22:03:59.364931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.364901 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.365056 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.365042 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.365099 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:03:59.365082 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:00.371427 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:00.371393 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerStarted","Data":"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4"} Oct 30 22:04:01.038368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:01.037447 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:04:01.051289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:01.050722 4525 patch_prober.go:29] interesting pod/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:01.051289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:01.050781 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:04:02.045635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:02.040195 4525 patch_prober.go:29] interesting pod/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.045635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:02.040270 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:04:02.805053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:02.805023 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:04:03.043305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:03.043268 4525 patch_prober.go:29] interesting pod/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.044091 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:03.044059 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:04:04.055764 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:04.055719 4525 patch_prober.go:29] interesting pod/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.056285 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:04.055799 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:04:05.047648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:05.046770 4525 patch_prober.go:29] interesting pod/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.047648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:05.046836 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 22:04:05.397994 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:04:05.397959 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:04:06.035598 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:06.035567 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:04:06.036782 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:06.036761 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:04:21.190122 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:21.190087 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 22:04:21.190694 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:21.190309 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" containerID="cri-o://7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5" gracePeriod=30 Oct 30 22:04:23.717458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:23.717420 4525 patch_prober.go:29] interesting pod/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:04:23.717910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:23.717478 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:04:26.477079 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.477019 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 22:04:26.571617 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571583 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.571812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571659 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.571812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571696 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.571928 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571852 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.571928 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571857 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.572046 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.571961 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.572046 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572009 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.572149 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572048 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.572149 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572105 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:26.572254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572155 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.572254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572237 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\") pod \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\" (UID: \"f0fcf030-0ddf-4cdc-96c5-ba642c790659\") " Oct 30 22:04:26.572254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572238 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.572411 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572283 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.572464 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.572449 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.573178 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.573136 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.573178 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.573168 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.573383 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.573184 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.573383 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.573198 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.573383 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.573213 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.574372 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.574349 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "kube-api-access-qxn8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:26.574500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.574381 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:26.574500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.574394 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:26.574500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.574453 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy" pod "f0fcf030-0ddf-4cdc-96c5-ba642c790659" (UID: "f0fcf030-0ddf-4cdc-96c5-ba642c790659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:26.668338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.668300 4525 generic.go:356] "Generic (PLEG): container finished" podID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerID="7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5" exitCode=0 Oct 30 22:04:26.668571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.668402 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" Oct 30 22:04:26.668571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.668426 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" event={"ID":"f0fcf030-0ddf-4cdc-96c5-ba642c790659","Type":"ContainerDied","Data":"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5"} Oct 30 22:04:26.668571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.668478 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2" event={"ID":"f0fcf030-0ddf-4cdc-96c5-ba642c790659","Type":"ContainerDied","Data":"191e06ea498610b1efb661154477cc575b82bbc80d6b7d18bbeae129727ed637"} Oct 30 22:04:26.668571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.668499 4525 scope.go:122] "RemoveContainer" containerID="7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5" Oct 30 22:04:26.674648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.674518 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn8q\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-kube-api-access-qxn8q\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.674648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.674550 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.674648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.674567 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.674648 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.674581 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fcf030-0ddf-4cdc-96c5-ba642c790659-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.680849 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.680826 4525 scope.go:122] "RemoveContainer" containerID="7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5" Oct 30 22:04:26.681165 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:04:26.681137 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5\": container with ID starting with 7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5 not found: ID does not exist" containerID="7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5" Oct 30 22:04:26.681265 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.681175 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5"} err="failed to get container status \"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5\": rpc error: code = NotFound desc = could not find container \"7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5\": container with ID starting with 7cd4faf035368694d34d0dddf14c374ad7d9872bdc467f5970d89e50d1fd3bc5 not found: ID does not exist" Oct 30 22:04:26.838058 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.837960 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 22:04:26.842633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:26.842573 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91874e-openshift-default-64b749d677-zbsf2"] Oct 30 22:04:27.477989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:27.477949 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" path="/var/lib/kubelet/pods/f0fcf030-0ddf-4cdc-96c5-ba642c790659/volumes" Oct 30 22:04:35.397970 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:04:35.397936 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:04:38.223467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.223429 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:04:38.224081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224067 4525 cpu_manager.go:414] "containerMap: removing container" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" Oct 30 22:04:38.224129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224082 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" Oct 30 22:04:38.224129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224099 4525 cpu_manager.go:414] "containerMap: removing container" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:04:38.224129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224104 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:04:38.224129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224125 4525 cpu_manager.go:414] "containerMap: removing container" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224130 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224144 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224149 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224268 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fcf030-0ddf-4cdc-96c5-ba642c790659" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224281 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:04:38.224289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224289 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="645f9f90-95b2-42e9-9808-c0db83424296" containerName="istio-proxy" Oct 30 22:04:38.224569 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224297 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:04:38.224569 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.224306 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bd8d19-7d8c-4c63-98cb-acdf3a4c59d4" containerName="istio-proxy" Oct 30 22:04:38.229266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.229248 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.283816 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.283423 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5bf4f8-serving-cert\"" Oct 30 22:04:38.287895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.287860 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:04:38.336188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.336141 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.336406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.336217 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.336406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.336321 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.336406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.336392 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.438122 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.438091 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.438341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.438145 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.438341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.438179 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.438341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.438196 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.438532 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:04:38.438364 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-serving-cert: secret "maas-api-e2e-api-b-5bf4f8-serving-cert" not found Oct 30 22:04:38.438532 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:04:38.438446 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls podName:2b4b053e-7637-4d7b-b88c-1e8bc3a796da nodeName:}" failed. No retries permitted until 2026-10-30 22:04:38.938426648 +0000 UTC m=+2503.612473183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls") pod "maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" (UID: "2b4b053e-7637-4d7b-b88c-1e8bc3a796da") : secret "maas-api-e2e-api-b-5bf4f8-serving-cert" not found Oct 30 22:04:38.439044 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.439020 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.440918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.440893 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.509865 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.509765 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.948053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.948003 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:38.950779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:38.950757 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") pod \"maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:39.143378 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:39.143341 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:39.529143 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:39.529037 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:04:39.536830 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:04:39.536795 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4b053e_7637_4d7b_b88c_1e8bc3a796da.slice/crio-220794c7e20d2f6edb54d7f497d5c96d493013070db37e4d59ce538a257ea96d WatchSource:0}: Error finding container 220794c7e20d2f6edb54d7f497d5c96d493013070db37e4d59ce538a257ea96d: Status 404 returned error can't find the container with id 220794c7e20d2f6edb54d7f497d5c96d493013070db37e4d59ce538a257ea96d Oct 30 22:04:39.538179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:39.538156 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:04:39.761278 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:39.761239 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" event={"ID":"2b4b053e-7637-4d7b-b88c-1e8bc3a796da","Type":"ContainerStarted","Data":"220794c7e20d2f6edb54d7f497d5c96d493013070db37e4d59ce538a257ea96d"} Oct 30 22:04:40.771664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:40.771537 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" event={"ID":"2b4b053e-7637-4d7b-b88c-1e8bc3a796da","Type":"ContainerStarted","Data":"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01"} Oct 30 22:04:40.771664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:40.771635 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:04:40.904561 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:40.904497 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" podStartSLOduration=2.044329203 podStartE2EDuration="2.904483143s" totalImagesPullingTime="860.15394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:40.90220578 +0000 UTC m=+2505.576252308" watchObservedRunningTime="2026-10-30 22:04:40.904483143 +0000 UTC m=+2505.578529712" Oct 30 22:04:40.912491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:40.912463 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:04:46.819434 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:04:46.819404 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:05:05.396956 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:05:05.396913 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:05:07.705846 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:07.705812 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 22:05:09.724235 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:09.724201 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 22:05:09.833560 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:09.833520 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 22:05:14.724590 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:14.724535 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" podUID="92d16570-6f18-4bba-862f-0303560cfe33" containerName="payload-processing" containerID="cri-o://32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2" gracePeriod=25 Oct 30 22:05:14.834232 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:14.834188 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" podUID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" containerName="payload-pre-processing" containerID="cri-o://3ae9fa17c5d842118b1015d544fc6cb76fefaab5ff450fb82a52a4467e24f1e7" gracePeriod=25 Oct 30 22:05:14.919748 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:14.919715 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 22:05:15.016994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.016968 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 22:05:15.055501 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.055469 4525 generic.go:356] "Generic (PLEG): container finished" podID="92d16570-6f18-4bba-862f-0303560cfe33" containerID="32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2" exitCode=0 Oct 30 22:05:15.055727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.055525 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" Oct 30 22:05:15.055727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.055558 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" event={"ID":"92d16570-6f18-4bba-862f-0303560cfe33","Type":"ContainerDied","Data":"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2"} Oct 30 22:05:15.055727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.055633 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv" event={"ID":"92d16570-6f18-4bba-862f-0303560cfe33","Type":"ContainerDied","Data":"991135cdeb78d1559e1bc8a6645fef294260d6ad38ff97400bdefef3214d3526"} Oct 30 22:05:15.055727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.055666 4525 scope.go:122] "RemoveContainer" containerID="32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2" Oct 30 22:05:15.057924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.057897 4525 generic.go:356] "Generic (PLEG): container finished" podID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" containerID="3ae9fa17c5d842118b1015d544fc6cb76fefaab5ff450fb82a52a4467e24f1e7" exitCode=0 Oct 30 22:05:15.058054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.057975 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" event={"ID":"1af2106e-af7b-4c0b-a81a-ffd125a99e06","Type":"ContainerDied","Data":"3ae9fa17c5d842118b1015d544fc6cb76fefaab5ff450fb82a52a4467e24f1e7"} Oct 30 22:05:15.076536 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.076503 4525 scope.go:122] "RemoveContainer" containerID="32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2" Oct 30 22:05:15.077114 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:05:15.077079 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2\": container with ID starting with 32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2 not found: ID does not exist" containerID="32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2" Oct 30 22:05:15.077226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.077118 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2"} err="failed to get container status \"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2\": rpc error: code = NotFound desc = could not find container \"32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2\": container with ID starting with 32bc763ca2979aa7cfe2105467a4d3db54a380e21742583b46352616d0dcdac2 not found: ID does not exist" Oct 30 22:05:15.133559 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.133529 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\" (UniqueName: \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\") pod \"92d16570-6f18-4bba-862f-0303560cfe33\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " Oct 30 22:05:15.133735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.133682 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\") pod \"92d16570-6f18-4bba-862f-0303560cfe33\" (UID: \"92d16570-6f18-4bba-862f-0303560cfe33\") " Oct 30 22:05:15.139352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.138524 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw" pod "92d16570-6f18-4bba-862f-0303560cfe33" (UID: "92d16570-6f18-4bba-862f-0303560cfe33"). InnerVolumeSpecName "kube-api-access-wrxlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:15.140983 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.140946 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume" pod "92d16570-6f18-4bba-862f-0303560cfe33" (UID: "92d16570-6f18-4bba-862f-0303560cfe33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:15.146963 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.146941 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 22:05:15.238054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.237962 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\") pod \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " Oct 30 22:05:15.238238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.238123 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\" (UniqueName: \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\") pod \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\" (UID: \"1af2106e-af7b-4c0b-a81a-ffd125a99e06\") " Oct 30 22:05:15.238735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.238709 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d16570-6f18-4bba-862f-0303560cfe33-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:15.238735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.238737 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxlw\" (UniqueName: \"kubernetes.io/projected/92d16570-6f18-4bba-862f-0303560cfe33-kube-api-access-wrxlw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:15.239446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.239419 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume" pod "1af2106e-af7b-4c0b-a81a-ffd125a99e06" (UID: "1af2106e-af7b-4c0b-a81a-ffd125a99e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:15.241341 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.241315 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd" pod "1af2106e-af7b-4c0b-a81a-ffd125a99e06" (UID: "1af2106e-af7b-4c0b-a81a-ffd125a99e06"). InnerVolumeSpecName "kube-api-access-b6zpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:15.390661 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.388657 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zpd\" (UniqueName: \"kubernetes.io/projected/1af2106e-af7b-4c0b-a81a-ffd125a99e06-kube-api-access-b6zpd\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:15.390661 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.388707 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af2106e-af7b-4c0b-a81a-ffd125a99e06-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:15.529961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.529933 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 22:05:15.534932 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:15.534905 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c5350-6f47865d4-6kkqv"] Oct 30 22:05:16.067126 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:16.067095 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" Oct 30 22:05:16.067126 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:16.067114 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x" event={"ID":"1af2106e-af7b-4c0b-a81a-ffd125a99e06","Type":"ContainerDied","Data":"980d1bcec8a1d07b6b1864d77bc70673982a0cb5ac5779803ff0d07b577aa705"} Oct 30 22:05:16.067699 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:16.067184 4525 scope.go:122] "RemoveContainer" containerID="3ae9fa17c5d842118b1015d544fc6cb76fefaab5ff450fb82a52a4467e24f1e7" Oct 30 22:05:16.235810 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:16.235772 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 22:05:16.240448 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:16.240420 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c5350-69cdbf585f-g677x"] Oct 30 22:05:17.475590 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:17.475555 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" path="/var/lib/kubelet/pods/1af2106e-af7b-4c0b-a81a-ffd125a99e06/volumes" Oct 30 22:05:17.476054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:17.476038 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d16570-6f18-4bba-862f-0303560cfe33" path="/var/lib/kubelet/pods/92d16570-6f18-4bba-862f-0303560cfe33/volumes" Oct 30 22:05:19.921055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:19.920984 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" containerID="cri-o://a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e" gracePeriod=25 Oct 30 22:05:22.623190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:22.623152 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:05:22.727455 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:22.727416 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" containerID="cri-o://d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e" gracePeriod=45 Oct 30 22:05:22.988323 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:22.988299 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 22:05:23.126528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126495 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.126787 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126594 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.126787 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126659 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.126787 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126725 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\" (UniqueName: \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.126787 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126788 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.127029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.126829 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\") pod \"7e263cd1-b190-49ec-8564-f0f5171d8a12\" (UID: \"7e263cd1-b190-49ec-8564-f0f5171d8a12\") " Oct 30 22:05:23.127530 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.127502 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.130331 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.130072 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.132306 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.132144 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:23.132683 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.132429 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.138331 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.134868 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:23.140384 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.140350 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd" pod "7e263cd1-b190-49ec-8564-f0f5171d8a12" (UID: "7e263cd1-b190-49ec-8564-f0f5171d8a12"). InnerVolumeSpecName "kube-api-access-lhdmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:23.140884 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.140856 4525 generic.go:356] "Generic (PLEG): container finished" podID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerID="d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e" exitCode=0 Oct 30 22:05:23.141213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.141191 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" event={"ID":"7e263cd1-b190-49ec-8564-f0f5171d8a12","Type":"ContainerDied","Data":"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e"} Oct 30 22:05:23.141282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.141238 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" event={"ID":"7e263cd1-b190-49ec-8564-f0f5171d8a12","Type":"ContainerDied","Data":"0d3335df9731d5eeec930181304a99495bd678aad9e6c7a78d66a3a8696e2aa1"} Oct 30 22:05:23.141282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.141260 4525 scope.go:122] "RemoveContainer" containerID="d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e" Oct 30 22:05:23.141461 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.141448 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" Oct 30 22:05:23.190290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.189879 4525 scope.go:122] "RemoveContainer" containerID="d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e" Oct 30 22:05:23.190290 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:05:23.190208 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e\": container with ID starting with d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e not found: ID does not exist" containerID="d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e" Oct 30 22:05:23.190290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.190254 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e"} err="failed to get container status \"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e\": rpc error: code = NotFound desc = could not find container \"d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e\": container with ID starting with d41cb68ec9811bb331df7c320493ccef0a9ccbf5cc08fc1f0019065c0cd9b71e not found: ID does not exist" Oct 30 22:05:23.228687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228649 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhdmd\" (UniqueName: \"kubernetes.io/projected/7e263cd1-b190-49ec-8564-f0f5171d8a12-kube-api-access-lhdmd\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.228687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228689 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.228882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228705 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.228882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228719 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.228882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228732 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e263cd1-b190-49ec-8564-f0f5171d8a12-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.228882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.228745 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e263cd1-b190-49ec-8564-f0f5171d8a12-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:23.313089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.313050 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 22:05:23.321467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.321426 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d"] Oct 30 22:05:23.474296 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.474221 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" path="/var/lib/kubelet/pods/7e263cd1-b190-49ec-8564-f0f5171d8a12/volumes" Oct 30 22:05:23.926218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:23.926171 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d68156/premium-w3-d68156-kserve-6865c485-jnm7d" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 22:05:25.425435 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:25.425393 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:35.407070 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:05:35.407011 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:05:35.425583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:35.425546 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:05:40.214129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.214103 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 22:05:40.312823 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.312793 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\") pod \"230f0a9e-a519-4e89-8567-72086ee4c551\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " Oct 30 22:05:40.313027 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.312858 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\" (UniqueName: \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\") pod \"230f0a9e-a519-4e89-8567-72086ee4c551\" (UID: \"230f0a9e-a519-4e89-8567-72086ee4c551\") " Oct 30 22:05:40.313179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.313155 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume" pod "230f0a9e-a519-4e89-8567-72086ee4c551" (UID: "230f0a9e-a519-4e89-8567-72086ee4c551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:40.313368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.313354 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f0a9e-a519-4e89-8567-72086ee4c551-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:40.315084 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.315062 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2" pod "230f0a9e-a519-4e89-8567-72086ee4c551" (UID: "230f0a9e-a519-4e89-8567-72086ee4c551"). InnerVolumeSpecName "kube-api-access-7f9l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:40.393716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.393675 4525 generic.go:356] "Generic (PLEG): container finished" podID="230f0a9e-a519-4e89-8567-72086ee4c551" containerID="a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e" exitCode=0 Oct 30 22:05:40.393907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.393726 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" Oct 30 22:05:40.393907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.393770 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" event={"ID":"230f0a9e-a519-4e89-8567-72086ee4c551","Type":"ContainerDied","Data":"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e"} Oct 30 22:05:40.393907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.393855 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5" event={"ID":"230f0a9e-a519-4e89-8567-72086ee4c551","Type":"ContainerDied","Data":"f87abfe2af0d9af89addfb4b6dc9655e2417048aff27871a2e6d133e17590ccd"} Oct 30 22:05:40.393907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.393886 4525 scope.go:122] "RemoveContainer" containerID="a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e" Oct 30 22:05:40.405083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.405066 4525 scope.go:122] "RemoveContainer" containerID="a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e" Oct 30 22:05:40.405383 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:05:40.405360 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e\": container with ID starting with a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e not found: ID does not exist" containerID="a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e" Oct 30 22:05:40.405437 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.405393 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e"} err="failed to get container status \"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e\": rpc error: code = NotFound desc = could not find container \"a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e\": container with ID starting with a4fe78245e49c79b21bc77302f2c6e763873d3fde6cba4c9a03dc6ab58fe351e not found: ID does not exist" Oct 30 22:05:40.414945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.414919 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9l2\" (UniqueName: \"kubernetes.io/projected/230f0a9e-a519-4e89-8567-72086ee4c551-kube-api-access-7f9l2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:40.560153 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.560071 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 22:05:40.565923 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:40.565898 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d68156-7c6459cb44-d5xn5"] Oct 30 22:05:41.475800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:41.475768 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" path="/var/lib/kubelet/pods/230f0a9e-a519-4e89-8567-72086ee4c551/volumes" Oct 30 22:05:48.266543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:48.266504 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 22:05:48.267051 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:48.266758 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" containerID="cri-o://6d3e71d3dde256acf4d5fcc4635da1323cc6eacf1e5e0bb448527a2439e118c9" gracePeriod=30 Oct 30 22:05:53.517133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.517099 4525 generic.go:356] "Generic (PLEG): container finished" podID="8869ab31-ec35-487b-912f-76c5a07430ac" containerID="6d3e71d3dde256acf4d5fcc4635da1323cc6eacf1e5e0bb448527a2439e118c9" exitCode=0 Oct 30 22:05:53.517571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.517149 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerDied","Data":"6d3e71d3dde256acf4d5fcc4635da1323cc6eacf1e5e0bb448527a2439e118c9"} Oct 30 22:05:53.517571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.517214 4525 scope.go:122] "RemoveContainer" containerID="2d6b9e999940bfb14a2756f4dc56ba0c731af187bd701b4f4769c71627577614" Oct 30 22:05:53.576549 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.576521 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 22:05:53.620535 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620503 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.620535 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620545 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.620828 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620809 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.620897 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620881 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.620988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620964 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:53.621072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.620973 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.621133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621096 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.621133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621121 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:53.621251 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621139 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.621251 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621181 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:53.621251 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621209 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.621417 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621305 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:53.621417 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.621314 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\") pod \"8869ab31-ec35-487b-912f-76c5a07430ac\" (UID: \"8869ab31-ec35-487b-912f-76c5a07430ac\") " Oct 30 22:05:53.622088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.622030 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.622088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.622066 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.622088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.622080 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.622374 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.622092 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.622374 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.622185 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:53.623368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.623343 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:53.623368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.623352 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "kube-api-access-gzgf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:53.624003 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.623982 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:53.624331 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.624309 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo" pod "8869ab31-ec35-487b-912f-76c5a07430ac" (UID: "8869ab31-ec35-487b-912f-76c5a07430ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:53.735631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.727875 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgf8\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-kube-api-access-gzgf8\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.735631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.727917 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8869ab31-ec35-487b-912f-76c5a07430ac-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.735631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.727932 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8869ab31-ec35-487b-912f-76c5a07430ac-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.735631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.727945 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8869ab31-ec35-487b-912f-76c5a07430ac-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:53.735631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:53.727958 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8869ab31-ec35-487b-912f-76c5a07430ac-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:54.526196 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:54.526167 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" Oct 30 22:05:54.526707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:54.526192 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h" event={"ID":"8869ab31-ec35-487b-912f-76c5a07430ac","Type":"ContainerDied","Data":"82c7a6e0bc0f5425475e9b7352b430e2fb3510e5b0cf8ed64bef82dfd2b0a9fc"} Oct 30 22:05:54.526707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:54.526258 4525 scope.go:122] "RemoveContainer" containerID="6d3e71d3dde256acf4d5fcc4635da1323cc6eacf1e5e0bb448527a2439e118c9" Oct 30 22:05:54.742599 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:54.742566 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 22:05:54.792230 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:54.792150 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-691598-openshift-default-c4b45464f-qkg4h"] Oct 30 22:05:55.476091 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:05:55.476054 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" path="/var/lib/kubelet/pods/8869ab31-ec35-487b-912f-76c5a07430ac/volumes" Oct 30 22:06:05.397036 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:05.396991 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:06:24.389421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:24.389391 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:06:24.503045 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:24.503016 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:06:29.382522 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.382485 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:06:29.389931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.389893 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" podUID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" containerName="payload-processing" containerID="cri-o://6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf" gracePeriod=25 Oct 30 22:06:29.503579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.503541 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" podUID="de248853-e91d-4c1d-b5d2-ff05662c6d13" containerName="payload-pre-processing" containerID="cri-o://f35c62589a9bc47262dabfccea64e4769191746bfb73686aa6e81d5858e390e1" gracePeriod=25 Oct 30 22:06:29.672952 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.672924 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:06:29.763127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.763013 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\") pod \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " Oct 30 22:06:29.763127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.763093 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\" (UniqueName: \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\") pod \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\" (UID: \"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8\") " Oct 30 22:06:29.763913 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.763866 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume" pod "5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" (UID: "5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:29.772723 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.770142 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p" pod "5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" (UID: "5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8"). InnerVolumeSpecName "kube-api-access-md49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:29.783725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.782330 4525 generic.go:356] "Generic (PLEG): container finished" podID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" containerID="6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf" exitCode=0 Oct 30 22:06:29.783725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.782448 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" event={"ID":"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8","Type":"ContainerDied","Data":"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf"} Oct 30 22:06:29.783725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.782500 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" event={"ID":"5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8","Type":"ContainerDied","Data":"fd5a10d2f44e24b344a80fe6285725a85f610cf5a4c280647e29511eff52f7c2"} Oct 30 22:06:29.783725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.782531 4525 scope.go:122] "RemoveContainer" containerID="6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf" Oct 30 22:06:29.783725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.782731 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk" Oct 30 22:06:29.786325 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.786296 4525 generic.go:356] "Generic (PLEG): container finished" podID="de248853-e91d-4c1d-b5d2-ff05662c6d13" containerID="f35c62589a9bc47262dabfccea64e4769191746bfb73686aa6e81d5858e390e1" exitCode=0 Oct 30 22:06:29.786444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.786402 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" event={"ID":"de248853-e91d-4c1d-b5d2-ff05662c6d13","Type":"ContainerDied","Data":"f35c62589a9bc47262dabfccea64e4769191746bfb73686aa6e81d5858e390e1"} Oct 30 22:06:29.830193 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.830160 4525 scope.go:122] "RemoveContainer" containerID="6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf" Oct 30 22:06:29.831199 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:29.831112 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf\": container with ID starting with 6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf not found: ID does not exist" containerID="6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf" Oct 30 22:06:29.831199 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.831151 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf"} err="failed to get container status \"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf\": rpc error: code = NotFound desc = could not find container \"6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf\": container with ID starting with 6de45ede1792662dbc6fc5b7ef0dcf7d9af54900486c3480fc4316e0f657a3bf not found: ID does not exist" Oct 30 22:06:29.872695 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.872581 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:29.872695 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.872631 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md49p\" (UniqueName: \"kubernetes.io/projected/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8-kube-api-access-md49p\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:29.881272 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.881247 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:06:29.959855 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.959780 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:06:29.966687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.966660 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98d38f-6d4d8c7bc8-g2zgk"] Oct 30 22:06:29.973462 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.973439 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\" (UniqueName: \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\") pod \"de248853-e91d-4c1d-b5d2-ff05662c6d13\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " Oct 30 22:06:29.973583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.973554 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\") pod \"de248853-e91d-4c1d-b5d2-ff05662c6d13\" (UID: \"de248853-e91d-4c1d-b5d2-ff05662c6d13\") " Oct 30 22:06:29.973926 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.973905 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume" pod "de248853-e91d-4c1d-b5d2-ff05662c6d13" (UID: "de248853-e91d-4c1d-b5d2-ff05662c6d13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:29.974168 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.974152 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de248853-e91d-4c1d-b5d2-ff05662c6d13-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:29.975790 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:29.975772 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d" pod "de248853-e91d-4c1d-b5d2-ff05662c6d13" (UID: "de248853-e91d-4c1d-b5d2-ff05662c6d13"). InnerVolumeSpecName "kube-api-access-4fk8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:30.075474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:30.075442 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fk8d\" (UniqueName: \"kubernetes.io/projected/de248853-e91d-4c1d-b5d2-ff05662c6d13-kube-api-access-4fk8d\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:30.855345 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:30.855243 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" event={"ID":"de248853-e91d-4c1d-b5d2-ff05662c6d13","Type":"ContainerDied","Data":"fec091a17b699e09dace7598ae95ae0475876465010f19dd270632d5da8dfe41"} Oct 30 22:06:30.855828 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:30.855374 4525 scope.go:122] "RemoveContainer" containerID="f35c62589a9bc47262dabfccea64e4769191746bfb73686aa6e81d5858e390e1" Oct 30 22:06:30.855828 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:30.855638 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m" Oct 30 22:06:31.057390 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:31.057357 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:06:31.130674 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:31.130388 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98d38f-5d7f748c7f-vgs6m"] Oct 30 22:06:31.474281 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:31.474206 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" path="/var/lib/kubelet/pods/5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8/volumes" Oct 30 22:06:31.474670 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:31.474655 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de248853-e91d-4c1d-b5d2-ff05662c6d13" path="/var/lib/kubelet/pods/de248853-e91d-4c1d-b5d2-ff05662c6d13/volumes" Oct 30 22:06:34.384308 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.384268 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" podUID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" containerName="payload-pre-processing" containerID="cri-o://380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea" gracePeriod=25 Oct 30 22:06:34.670891 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.670863 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:06:34.739863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.739830 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\" (UniqueName: \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\") pod \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " Oct 30 22:06:34.740089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.740076 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\") pod \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\" (UID: \"8f64dfc8-9934-478c-8317-8f301f0f1b2e\") " Oct 30 22:06:34.740449 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.740423 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume" pod "8f64dfc8-9934-478c-8317-8f301f0f1b2e" (UID: "8f64dfc8-9934-478c-8317-8f301f0f1b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:34.742280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.742256 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj" pod "8f64dfc8-9934-478c-8317-8f301f0f1b2e" (UID: "8f64dfc8-9934-478c-8317-8f301f0f1b2e"). InnerVolumeSpecName "kube-api-access-2qzqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:34.842020 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.841980 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f64dfc8-9934-478c-8317-8f301f0f1b2e-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:34.842020 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.842016 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzqj\" (UniqueName: \"kubernetes.io/projected/8f64dfc8-9934-478c-8317-8f301f0f1b2e-kube-api-access-2qzqj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:34.883945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.883906 4525 generic.go:356] "Generic (PLEG): container finished" podID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" containerID="380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea" exitCode=0 Oct 30 22:06:34.884115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.883955 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" Oct 30 22:06:34.884115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.883989 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" event={"ID":"8f64dfc8-9934-478c-8317-8f301f0f1b2e","Type":"ContainerDied","Data":"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea"} Oct 30 22:06:34.884115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.884051 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2" event={"ID":"8f64dfc8-9934-478c-8317-8f301f0f1b2e","Type":"ContainerDied","Data":"8d08797954f265f308de6a61c09825e1a1fa17bc9b1f38b081e0179a31ac045e"} Oct 30 22:06:34.884115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.884075 4525 scope.go:122] "RemoveContainer" containerID="380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea" Oct 30 22:06:34.894343 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.894326 4525 scope.go:122] "RemoveContainer" containerID="380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea" Oct 30 22:06:34.894666 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:34.894645 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea\": container with ID starting with 380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea not found: ID does not exist" containerID="380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea" Oct 30 22:06:34.894717 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:34.894675 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea"} err="failed to get container status \"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea\": rpc error: code = NotFound desc = could not find container \"380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea\": container with ID starting with 380bb7e7b65f7c5e4a40c42ba975f5b62df70db942119628d6bfdc429de08cea not found: ID does not exist" Oct 30 22:06:35.051022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:35.050957 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:06:35.054183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:35.054156 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ae1bd5-754f856999-vmhf2"] Oct 30 22:06:35.395890 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:35.395855 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:06:35.477239 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:35.477205 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" path="/var/lib/kubelet/pods/8f64dfc8-9934-478c-8317-8f301f0f1b2e/volumes" Oct 30 22:06:41.127187 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:41.127148 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:06:41.127587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:41.127442 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" containerID="cri-o://7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4" gracePeriod=30 Oct 30 22:06:41.654077 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:41.654039 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:06:46.393338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.393278 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:06:46.506167 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506136 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506226 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506252 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506300 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506371 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506405 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506494 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506724 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506563 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506724 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506650 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\") pod \"7ff303c6-980d-4f0a-85f7-c632f60543b6\" (UID: \"7ff303c6-980d-4f0a-85f7-c632f60543b6\") " Oct 30 22:06:46.506724 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506686 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:46.506892 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506715 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:46.506892 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506778 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:46.507008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.506889 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507256 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507512 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507535 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507550 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507568 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ff303c6-980d-4f0a-85f7-c632f60543b6-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.507581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.507582 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.509043 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.509003 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "kube-api-access-wqbg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:46.509186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.509063 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:46.509435 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.509416 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:06:46.509502 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.509435 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy" pod "7ff303c6-980d-4f0a-85f7-c632f60543b6" (UID: "7ff303c6-980d-4f0a-85f7-c632f60543b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:46.609099 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.609054 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.609099 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.609096 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbg5\" (UniqueName: \"kubernetes.io/projected/7ff303c6-980d-4f0a-85f7-c632f60543b6-kube-api-access-wqbg5\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.609099 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.609108 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.609099 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.609117 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ff303c6-980d-4f0a-85f7-c632f60543b6-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:46.967696 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.967661 4525 generic.go:356] "Generic (PLEG): container finished" podID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerID="7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4" exitCode=0 Oct 30 22:06:46.967902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.967715 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" Oct 30 22:06:46.967902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.967743 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerDied","Data":"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4"} Oct 30 22:06:46.967902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.967806 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t" event={"ID":"7ff303c6-980d-4f0a-85f7-c632f60543b6","Type":"ContainerDied","Data":"c0d1e22329f1c969b421b0d6ca7006dcef4c0916e20078d6377c4b689b5abdf5"} Oct 30 22:06:46.967902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.967836 4525 scope.go:122] "RemoveContainer" containerID="7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4" Oct 30 22:06:46.978450 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.978430 4525 scope.go:122] "RemoveContainer" containerID="fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08" Oct 30 22:06:46.987711 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.987690 4525 scope.go:122] "RemoveContainer" containerID="7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4" Oct 30 22:06:46.988021 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:46.988003 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4\": container with ID starting with 7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4 not found: ID does not exist" containerID="7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4" Oct 30 22:06:46.988080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.988030 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4"} err="failed to get container status \"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4\": rpc error: code = NotFound desc = could not find container \"7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4\": container with ID starting with 7497be9c7d089891568189c93c10dcb5f3fad7438bf61b30832322c700f971e4 not found: ID does not exist" Oct 30 22:06:46.988080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.988048 4525 scope.go:122] "RemoveContainer" containerID="fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08" Oct 30 22:06:46.988299 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:06:46.988278 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08\": container with ID starting with fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08 not found: ID does not exist" containerID="fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08" Oct 30 22:06:46.988359 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:46.988309 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08"} err="failed to get container status \"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08\": rpc error: code = NotFound desc = could not find container \"fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08\": container with ID starting with fd8f454a80946e3f2c4bbd4103b4d9710f93f36f055019ee8f02ba56fef05b08 not found: ID does not exist" Oct 30 22:06:47.134651 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:47.134618 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:06:47.140733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:47.140708 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-abb526-openshift-default-7d4b956cbf-9cs6t"] Oct 30 22:06:47.474431 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:47.474399 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" path="/var/lib/kubelet/pods/7ff303c6-980d-4f0a-85f7-c632f60543b6/volumes" Oct 30 22:06:51.743363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743332 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:06:51.743961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743946 4525 cpu_manager.go:414] "containerMap: removing container" podUID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" containerName="payload-processing" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743962 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" containerName="payload-processing" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743981 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743986 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.743998 4525 cpu_manager.go:414] "containerMap: removing container" podUID="92d16570-6f18-4bba-862f-0303560cfe33" containerName="payload-processing" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744003 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d16570-6f18-4bba-862f-0303560cfe33" containerName="payload-processing" Oct 30 22:06:51.744009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744009 4525 cpu_manager.go:414] "containerMap: removing container" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744014 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744026 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744030 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744045 4525 cpu_manager.go:414] "containerMap: removing container" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744050 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f06d1d-ae5a-4f0d-9f27-45436aeb37ce" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744066 4525 cpu_manager.go:414] "containerMap: removing container" podUID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744070 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744084 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744089 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744095 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744100 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744109 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744114 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744120 4525 cpu_manager.go:414] "containerMap: removing container" podUID="de248853-e91d-4c1d-b5d2-ff05662c6d13" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744124 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de248853-e91d-4c1d-b5d2-ff05662c6d13" containerName="payload-pre-processing" Oct 30 22:06:51.744195 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744200 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af2106e-af7b-4c0b-a81a-ffd125a99e06" containerName="payload-pre-processing" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744210 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744217 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e263cd1-b190-49ec-8564-f0f5171d8a12" containerName="main" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744224 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8869ab31-ec35-487b-912f-76c5a07430ac" containerName="istio-proxy" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744230 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="de248853-e91d-4c1d-b5d2-ff05662c6d13" containerName="payload-pre-processing" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744237 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5860e9c8-31f9-4bbd-9ddc-cfb4f48014a8" containerName="payload-processing" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744245 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744251 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744258 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d16570-6f18-4bba-862f-0303560cfe33" containerName="payload-processing" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744264 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="230f0a9e-a519-4e89-8567-72086ee4c551" containerName="payload-processing" Oct 30 22:06:51.744707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.744270 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f64dfc8-9934-478c-8317-8f301f0f1b2e" containerName="payload-pre-processing" Oct 30 22:06:51.748957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.748938 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.844464 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.844431 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6dcc0\"/\"kube-root-ca.crt\"" Oct 30 22:06:51.844664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.844496 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e6dcc0\"/\"openshift-service-ca.crt\"" Oct 30 22:06:51.844664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.844499 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6dcc0\"/\"default-dockercfg-9kzvb\"" Oct 30 22:06:51.853026 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.852996 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e6dcc0\"/\"test-model-e6dcc0-kserve-self-signed-certs\"" Oct 30 22:06:51.856126 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.856099 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:06:51.875781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.875752 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.875957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.875820 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.875957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.875936 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj5g\" (UniqueName: \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.876086 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.875998 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.876086 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.876064 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.876202 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.876107 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977269 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977240 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977323 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977362 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj5g\" (UniqueName: \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977414 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977444 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977484 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977828 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977913 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977882 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.977949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.977920 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.979925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.979893 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:51.980050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:51.980032 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:52.039503 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:52.039428 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj5g\" (UniqueName: \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\") pod \"test-model-e6dcc0-kserve-99b56cdb9-46fnd\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:52.062635 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:52.062572 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:52.445447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:52.445413 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:06:52.446318 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:06:52.446291 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c2f241c_8757_4636_a978_34ef53274d57.slice/crio-725536e2a73f6b37c4d72bbf78de9bf50b2c6d48506c115910c14f6b39629e16 WatchSource:0}: Error finding container 725536e2a73f6b37c4d72bbf78de9bf50b2c6d48506c115910c14f6b39629e16: Status 404 returned error can't find the container with id 725536e2a73f6b37c4d72bbf78de9bf50b2c6d48506c115910c14f6b39629e16 Oct 30 22:06:53.015985 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:53.015943 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" event={"ID":"3c2f241c-8757-4636-a978-34ef53274d57","Type":"ContainerStarted","Data":"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06"} Oct 30 22:06:53.016495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:53.016416 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" event={"ID":"3c2f241c-8757-4636-a978-34ef53274d57","Type":"ContainerStarted","Data":"725536e2a73f6b37c4d72bbf78de9bf50b2c6d48506c115910c14f6b39629e16"} Oct 30 22:06:53.063356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:53.063319 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:06:53.134705 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:53.134647 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" podStartSLOduration=2.1345992799999998 podStartE2EDuration="2.13459928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.133969876 +0000 UTC m=+2637.808016414" watchObservedRunningTime="2026-10-30 22:06:53.13459928 +0000 UTC m=+2637.808645855" Oct 30 22:06:59.040985 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.040953 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 22:06:59.041453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.041179 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" containerID="cri-o://0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049" gracePeriod=30 Oct 30 22:06:59.360051 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.360020 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 22:06:59.493633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.493280 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\") pod \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " Oct 30 22:06:59.493633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.493370 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\" (UniqueName: \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\") pod \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " Oct 30 22:06:59.493633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.493468 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") pod \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " Oct 30 22:06:59.493633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.493594 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\") pod \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\" (UID: \"d294fa8d-5d61-4df5-afd5-570eab9a0c6f\") " Oct 30 22:06:59.494777 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.494683 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca" pod "d294fa8d-5d61-4df5-afd5-570eab9a0c6f" (UID: "d294fa8d-5d61-4df5-afd5-570eab9a0c6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:59.504636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.503045 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5" pod "d294fa8d-5d61-4df5-afd5-570eab9a0c6f" (UID: "d294fa8d-5d61-4df5-afd5-570eab9a0c6f"). InnerVolumeSpecName "kube-api-access-vxdd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:59.504636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.503781 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls" pod "d294fa8d-5d61-4df5-afd5-570eab9a0c6f" (UID: "d294fa8d-5d61-4df5-afd5-570eab9a0c6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:59.505106 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.505032 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs" pod "d294fa8d-5d61-4df5-afd5-570eab9a0c6f" (UID: "d294fa8d-5d61-4df5-afd5-570eab9a0c6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:06:59.604712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.602471 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:59.604712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.602502 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:59.604712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.602515 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdd5\" (UniqueName: \"kubernetes.io/projected/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-kube-api-access-vxdd5\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:59.604712 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:06:59.602525 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d294fa8d-5d61-4df5-afd5-570eab9a0c6f-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:00.148627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.137350 4525 generic.go:356] "Generic (PLEG): container finished" podID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerID="0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049" exitCode=0 Oct 30 22:07:00.148627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.137433 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerDied","Data":"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049"} Oct 30 22:07:00.148627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.137484 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" event={"ID":"d294fa8d-5d61-4df5-afd5-570eab9a0c6f","Type":"ContainerDied","Data":"2b23765ecbc7c43e1f40e6cb22cd8849704e0c599ff42bec4e7598f4f01673da"} Oct 30 22:07:00.148627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.137514 4525 scope.go:122] "RemoveContainer" containerID="0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049" Oct 30 22:07:00.148627 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.137748 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc" Oct 30 22:07:00.165337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.165292 4525 scope.go:122] "RemoveContainer" containerID="d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df" Oct 30 22:07:00.175569 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.175550 4525 scope.go:122] "RemoveContainer" containerID="0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049" Oct 30 22:07:00.175972 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:07:00.175920 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049\": container with ID starting with 0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049 not found: ID does not exist" containerID="0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049" Oct 30 22:07:00.175972 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.175971 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049"} err="failed to get container status \"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049\": rpc error: code = NotFound desc = could not find container \"0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049\": container with ID starting with 0a304752d5668fb29b0cd9160f86a4951b225bd01585c0de54ead5e0b1bf4049 not found: ID does not exist" Oct 30 22:07:00.176227 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.175994 4525 scope.go:122] "RemoveContainer" containerID="d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df" Oct 30 22:07:00.176280 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:07:00.176268 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df\": container with ID starting with d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df not found: ID does not exist" containerID="d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df" Oct 30 22:07:00.176336 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.176288 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df"} err="failed to get container status \"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df\": rpc error: code = NotFound desc = could not find container \"d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df\": container with ID starting with d3b1449f5075ae7eecd07f395cdf027be4175fcc491ccf92d4e928e43d7261df not found: ID does not exist" Oct 30 22:07:00.318013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.317964 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 22:07:00.322288 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:00.322258 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d68156-749cf4d964-jz4vc"] Oct 30 22:07:01.473945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:01.473909 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" path="/var/lib/kubelet/pods/d294fa8d-5d61-4df5-afd5-570eab9a0c6f/volumes" Oct 30 22:07:02.063721 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:02.063687 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:07:02.078066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:02.078037 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:07:02.184973 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:02.184936 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:07:05.435289 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:07:05.435251 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:07:35.397872 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:07:35.397785 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:07:38.349305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:38.349265 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:07:56.877501 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:56.877460 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 22:07:57.015777 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:07:57.015737 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 22:08:01.878059 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:01.877958 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" podUID="491530a1-5be4-4429-a2ca-6299ce346b47" containerName="payload-processing" containerID="cri-o://fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924" gracePeriod=25 Oct 30 22:08:02.016851 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.016757 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" podUID="93b4a926-8cc2-4297-a68b-510d6e507366" containerName="payload-pre-processing" containerID="cri-o://de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346" gracePeriod=25 Oct 30 22:08:02.167253 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.167224 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 22:08:02.298277 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.298241 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 22:08:02.306882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.306855 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\") pod \"491530a1-5be4-4429-a2ca-6299ce346b47\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " Oct 30 22:08:02.307053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.306965 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\" (UniqueName: \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\") pod \"491530a1-5be4-4429-a2ca-6299ce346b47\" (UID: \"491530a1-5be4-4429-a2ca-6299ce346b47\") " Oct 30 22:08:02.307276 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.307254 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume" pod "491530a1-5be4-4429-a2ca-6299ce346b47" (UID: "491530a1-5be4-4429-a2ca-6299ce346b47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:02.307447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.307426 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491530a1-5be4-4429-a2ca-6299ce346b47-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:02.309273 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.309246 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h" pod "491530a1-5be4-4429-a2ca-6299ce346b47" (UID: "491530a1-5be4-4429-a2ca-6299ce346b47"). InnerVolumeSpecName "kube-api-access-zg42h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:02.408641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.408532 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\") pod \"93b4a926-8cc2-4297-a68b-510d6e507366\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " Oct 30 22:08:02.408813 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.408780 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\" (UniqueName: \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\") pod \"93b4a926-8cc2-4297-a68b-510d6e507366\" (UID: \"93b4a926-8cc2-4297-a68b-510d6e507366\") " Oct 30 22:08:02.408961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.408938 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume" pod "93b4a926-8cc2-4297-a68b-510d6e507366" (UID: "93b4a926-8cc2-4297-a68b-510d6e507366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:02.409181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.409167 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg42h\" (UniqueName: \"kubernetes.io/projected/491530a1-5be4-4429-a2ca-6299ce346b47-kube-api-access-zg42h\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:02.409256 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.409182 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b4a926-8cc2-4297-a68b-510d6e507366-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:02.411040 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.411017 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9" pod "93b4a926-8cc2-4297-a68b-510d6e507366" (UID: "93b4a926-8cc2-4297-a68b-510d6e507366"). InnerVolumeSpecName "kube-api-access-l75g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:02.520203 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.517549 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75g9\" (UniqueName: \"kubernetes.io/projected/93b4a926-8cc2-4297-a68b-510d6e507366-kube-api-access-l75g9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:02.674512 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.674374 4525 generic.go:356] "Generic (PLEG): container finished" podID="491530a1-5be4-4429-a2ca-6299ce346b47" containerID="fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924" exitCode=0 Oct 30 22:08:02.674512 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.674433 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" Oct 30 22:08:02.674512 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.674463 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" event={"ID":"491530a1-5be4-4429-a2ca-6299ce346b47","Type":"ContainerDied","Data":"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924"} Oct 30 22:08:02.674836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.674521 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm" event={"ID":"491530a1-5be4-4429-a2ca-6299ce346b47","Type":"ContainerDied","Data":"de80453c4a0a21b8073f669fe281f7795e4cb7e47e397ca1532eb9aa169419bf"} Oct 30 22:08:02.674836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.674571 4525 scope.go:122] "RemoveContainer" containerID="fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924" Oct 30 22:08:02.676280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.676257 4525 generic.go:356] "Generic (PLEG): container finished" podID="93b4a926-8cc2-4297-a68b-510d6e507366" containerID="de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346" exitCode=0 Oct 30 22:08:02.676409 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.676294 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" Oct 30 22:08:02.676409 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.676300 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" event={"ID":"93b4a926-8cc2-4297-a68b-510d6e507366","Type":"ContainerDied","Data":"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346"} Oct 30 22:08:02.676504 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.676440 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj" event={"ID":"93b4a926-8cc2-4297-a68b-510d6e507366","Type":"ContainerDied","Data":"8cb822e2616fa2f0018529fc33b9d59406404e017d4aec8121186c949ffbf770"} Oct 30 22:08:02.685260 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.685240 4525 scope.go:122] "RemoveContainer" containerID="fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924" Oct 30 22:08:02.685552 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:02.685527 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924\": container with ID starting with fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924 not found: ID does not exist" containerID="fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924" Oct 30 22:08:02.685623 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.685567 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924"} err="failed to get container status \"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924\": rpc error: code = NotFound desc = could not find container \"fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924\": container with ID starting with fc567ff8d2a50509946568c88ea0d32f2229f2b2179171db175e3de5ec82f924 not found: ID does not exist" Oct 30 22:08:02.685623 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.685592 4525 scope.go:122] "RemoveContainer" containerID="de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346" Oct 30 22:08:02.695519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.695504 4525 scope.go:122] "RemoveContainer" containerID="de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346" Oct 30 22:08:02.695832 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:02.695813 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346\": container with ID starting with de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346 not found: ID does not exist" containerID="de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346" Oct 30 22:08:02.695908 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.695839 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346"} err="failed to get container status \"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346\": rpc error: code = NotFound desc = could not find container \"de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346\": container with ID starting with de5b8e04c8a8ab83a8d29cd64bcd3a37cccf8e7de77341d8c0850a8a525d5346 not found: ID does not exist" Oct 30 22:08:02.842732 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.842694 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 22:08:02.892843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.892804 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1a415f-6c68cd8f58-s4ccm"] Oct 30 22:08:02.896048 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896024 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:08:02.896835 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896813 4525 cpu_manager.go:414] "containerMap: removing container" podUID="93b4a926-8cc2-4297-a68b-510d6e507366" containerName="payload-pre-processing" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896836 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b4a926-8cc2-4297-a68b-510d6e507366" containerName="payload-pre-processing" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896859 4525 cpu_manager.go:414] "containerMap: removing container" podUID="491530a1-5be4-4429-a2ca-6299ce346b47" containerName="payload-processing" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896867 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491530a1-5be4-4429-a2ca-6299ce346b47" containerName="payload-processing" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896910 4525 cpu_manager.go:414] "containerMap: removing container" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896917 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff303c6-980d-4f0a-85f7-c632f60543b6" containerName="istio-proxy" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896930 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.896937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896937 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896953 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.896961 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.897090 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.897103 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d294fa8d-5d61-4df5-afd5-570eab9a0c6f" containerName="maas-api" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.897118 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b4a926-8cc2-4297-a68b-510d6e507366" containerName="payload-pre-processing" Oct 30 22:08:02.897159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.897129 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="491530a1-5be4-4429-a2ca-6299ce346b47" containerName="payload-processing" Oct 30 22:08:02.901936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.901919 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:02.965000 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.964899 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd2a94-openshift-default-dockercfg-qmcvv\"" Oct 30 22:08:02.968535 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.968503 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 22:08:02.973410 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.973383 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:08:02.977578 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:02.977551 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1a415f-6694db77f9-txzmj"] Oct 30 22:08:03.023781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.023750 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.023949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.023787 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lq42\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.023949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.023906 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024038 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.023987 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024038 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.024030 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024117 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.024091 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024167 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.024155 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024264 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.024249 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.024303 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.024283 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.125698 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.125664 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.125918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.125725 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.125918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.125806 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.125918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.125902 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126103 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.125945 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126103 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126004 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126103 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126038 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lq42\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126103 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126093 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126148 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126206 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126567 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126542 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126565 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126572 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.126770 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.126753 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.128867 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.128846 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.129107 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.129088 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.187771 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.187736 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.187969 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.187950 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lq42\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\") pod \"e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.216581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.216498 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:03.474429 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.474356 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491530a1-5be4-4429-a2ca-6299ce346b47" path="/var/lib/kubelet/pods/491530a1-5be4-4429-a2ca-6299ce346b47/volumes" Oct 30 22:08:03.474848 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.474833 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b4a926-8cc2-4297-a68b-510d6e507366" path="/var/lib/kubelet/pods/93b4a926-8cc2-4297-a68b-510d6e507366/volumes" Oct 30 22:08:03.648497 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.648456 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:08:03.648996 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:08:03.648973 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4f334c_f3fd_4052_926f_c10c16bd7049.slice/crio-90c48d8f2f3e86627242e95fdff09018403e2e5a9d6550b5ef64d192fbe238f7 WatchSource:0}: Error finding container 90c48d8f2f3e86627242e95fdff09018403e2e5a9d6550b5ef64d192fbe238f7: Status 404 returned error can't find the container with id 90c48d8f2f3e86627242e95fdff09018403e2e5a9d6550b5ef64d192fbe238f7 Oct 30 22:08:03.651536 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.651503 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:03.651630 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.651570 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:03.651630 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.651599 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:03.684444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:03.684409 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" event={"ID":"ee4f334c-f3fd-4052-926f-c10c16bd7049","Type":"ContainerStarted","Data":"90c48d8f2f3e86627242e95fdff09018403e2e5a9d6550b5ef64d192fbe238f7"} Oct 30 22:08:04.693840 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:04.693803 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" event={"ID":"ee4f334c-f3fd-4052-926f-c10c16bd7049","Type":"ContainerStarted","Data":"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b"} Oct 30 22:08:04.871033 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:04.870979 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podStartSLOduration=2.870966226 podStartE2EDuration="2.870966226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:04.868915374 +0000 UTC m=+2709.542961904" watchObservedRunningTime="2026-10-30 22:08:04.870966226 +0000 UTC m=+2709.545012761" Oct 30 22:08:05.217357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:05.217329 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:05.219290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:05.219254 4525 patch_prober.go:29] interesting pod/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:08:05.219422 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:05.219304 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 22:08:05.396824 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:05.396778 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:08:06.217270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:06.217233 4525 patch_prober.go:29] interesting pod/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:08:06.217737 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:06.217290 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 22:08:07.217047 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:07.217012 4525 patch_prober.go:29] interesting pod/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:08:07.217332 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:07.217067 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 22:08:08.217002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:08.216970 4525 patch_prober.go:29] interesting pod/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:08:08.217199 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:08.217021 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 22:08:09.217677 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:09.217642 4525 patch_prober.go:29] interesting pod/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 22:08:09.218102 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:09.217704 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 22:08:10.222467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:10.222436 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:10.222908 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:10.222766 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:10.223470 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:10.223449 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:08:19.398563 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:19.398528 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:08:27.447556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.447523 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:08:27.448034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.447771 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" containerID="cri-o://fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48" gracePeriod=30 Oct 30 22:08:27.761438 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.761409 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:08:27.865887 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.865851 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") pod \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " Oct 30 22:08:27.866181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.866157 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\") pod \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " Oct 30 22:08:27.866362 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.866346 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\" (UniqueName: \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\") pod \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " Oct 30 22:08:27.866671 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.866584 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\") pod \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\" (UID: \"e03d13fc-4001-4c40-bfe7-ff6402a14ab2\") " Oct 30 22:08:27.869147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.869120 4525 generic.go:356] "Generic (PLEG): container finished" podID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerID="fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48" exitCode=0 Oct 30 22:08:27.869393 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.869375 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerDied","Data":"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48"} Oct 30 22:08:27.869561 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.869534 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" event={"ID":"e03d13fc-4001-4c40-bfe7-ff6402a14ab2","Type":"ContainerDied","Data":"1362fe1f8c25bddfdcdb5763fe677eeedaf157e6ca17f5c5009504b56747d0cc"} Oct 30 22:08:27.869703 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.869690 4525 scope.go:122] "RemoveContainer" containerID="fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48" Oct 30 22:08:27.869953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.869939 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm" Oct 30 22:08:27.881415 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.881370 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca" pod "e03d13fc-4001-4c40-bfe7-ff6402a14ab2" (UID: "e03d13fc-4001-4c40-bfe7-ff6402a14ab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:27.890867 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.890830 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v" pod "e03d13fc-4001-4c40-bfe7-ff6402a14ab2" (UID: "e03d13fc-4001-4c40-bfe7-ff6402a14ab2"). InnerVolumeSpecName "kube-api-access-bzk4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:27.895196 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.895143 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls" pod "e03d13fc-4001-4c40-bfe7-ff6402a14ab2" (UID: "e03d13fc-4001-4c40-bfe7-ff6402a14ab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:27.896690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.896658 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs" pod "e03d13fc-4001-4c40-bfe7-ff6402a14ab2" (UID: "e03d13fc-4001-4c40-bfe7-ff6402a14ab2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:27.923454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.923417 4525 scope.go:122] "RemoveContainer" containerID="9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5" Oct 30 22:08:27.933707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.933683 4525 scope.go:122] "RemoveContainer" containerID="fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48" Oct 30 22:08:27.934136 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:27.934100 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48\": container with ID starting with fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48 not found: ID does not exist" containerID="fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48" Oct 30 22:08:27.934253 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.934148 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48"} err="failed to get container status \"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48\": rpc error: code = NotFound desc = could not find container \"fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48\": container with ID starting with fb7055cc36ce2448812ca14845042957cd3d3468d438b4948aae2be9f21efa48 not found: ID does not exist" Oct 30 22:08:27.934253 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.934174 4525 scope.go:122] "RemoveContainer" containerID="9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5" Oct 30 22:08:27.934529 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:27.934507 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5\": container with ID starting with 9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5 not found: ID does not exist" containerID="9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5" Oct 30 22:08:27.934707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.934543 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5"} err="failed to get container status \"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5\": rpc error: code = NotFound desc = could not find container \"9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5\": container with ID starting with 9b2a8a516d644010a26f359421940f2c1f7d6f74bac821c9fc4c51ae238455b5 not found: ID does not exist" Oct 30 22:08:27.968732 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.968704 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:27.968732 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.968732 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:27.968907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.968742 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:27.968907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:27.968751 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzk4v\" (UniqueName: \"kubernetes.io/projected/e03d13fc-4001-4c40-bfe7-ff6402a14ab2-kube-api-access-bzk4v\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:28.295747 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:28.295707 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:08:28.350068 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:28.350028 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae1bd5-7f4955744c-5w4xm"] Oct 30 22:08:29.475240 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:29.475211 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" path="/var/lib/kubelet/pods/e03d13fc-4001-4c40-bfe7-ff6402a14ab2/volumes" Oct 30 22:08:31.922601 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:31.922563 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 22:08:31.923134 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:31.922841 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" containerID="cri-o://2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd" gracePeriod=30 Oct 30 22:08:35.397162 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:35.397115 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:08:37.210665 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.210640 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 22:08:37.301502 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301471 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301546 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301576 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301666 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301689 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301738 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301758 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301817 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.301956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301869 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:37.301956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301924 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\") pod \"6d5e7f17-3426-4d62-af1a-181e70840c70\" (UID: \"6d5e7f17-3426-4d62-af1a-181e70840c70\") " Oct 30 22:08:37.302239 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.301995 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:37.302239 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302193 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:37.302349 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302236 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:37.302416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302380 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:37.302643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302579 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.302643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302642 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.302643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302659 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.302643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302673 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5e7f17-3426-4d62-af1a-181e70840c70-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.302934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.302687 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.303987 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.303954 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:37.304147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.304124 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:37.304147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.304140 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "kube-api-access-dsbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:37.304366 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.304342 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo" pod "6d5e7f17-3426-4d62-af1a-181e70840c70" (UID: "6d5e7f17-3426-4d62-af1a-181e70840c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:37.404197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.404160 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbzh\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-kube-api-access-dsbzh\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.404197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.404198 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.404397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.404212 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.404397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.404224 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5e7f17-3426-4d62-af1a-181e70840c70-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:37.932624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.932566 4525 generic.go:356] "Generic (PLEG): container finished" podID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerID="2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd" exitCode=0 Oct 30 22:08:37.932820 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.932650 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerDied","Data":"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd"} Oct 30 22:08:37.932820 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.932674 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" Oct 30 22:08:37.932820 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.932760 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs" event={"ID":"6d5e7f17-3426-4d62-af1a-181e70840c70","Type":"ContainerDied","Data":"e69e0efe194d4e4a9f20f57ec14411620b59cb682d73e8806f5d2c58720646bb"} Oct 30 22:08:37.933008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.932848 4525 scope.go:122] "RemoveContainer" containerID="2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd" Oct 30 22:08:37.943987 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.943963 4525 scope.go:122] "RemoveContainer" containerID="fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f" Oct 30 22:08:37.954690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.954496 4525 scope.go:122] "RemoveContainer" containerID="2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd" Oct 30 22:08:37.957995 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:37.957970 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd\": container with ID starting with 2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd not found: ID does not exist" containerID="2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd" Oct 30 22:08:37.958135 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.958003 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd"} err="failed to get container status \"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd\": rpc error: code = NotFound desc = could not find container \"2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd\": container with ID starting with 2f17cdd768dba99c8b593bff1120a9efe2690b2b6666f9059ea617bb3737a7fd not found: ID does not exist" Oct 30 22:08:37.958135 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.958027 4525 scope.go:122] "RemoveContainer" containerID="fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f" Oct 30 22:08:37.958379 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:37.958360 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f\": container with ID starting with fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f not found: ID does not exist" containerID="fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f" Oct 30 22:08:37.958451 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:37.958384 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f"} err="failed to get container status \"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f\": rpc error: code = NotFound desc = could not find container \"fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f\": container with ID starting with fac746dec694c51cc053c696e9b9db39d6a617cf8930d204e0991c7f3e9db59f not found: ID does not exist" Oct 30 22:08:38.028297 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:38.028254 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:08:38.028538 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:38.028511 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" containerID="cri-o://d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027" gracePeriod=30 Oct 30 22:08:38.176688 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:38.176651 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 22:08:38.202821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:38.202731 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-bdkzs"] Oct 30 22:08:39.474943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:39.474906 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" path="/var/lib/kubelet/pods/6d5e7f17-3426-4d62-af1a-181e70840c70/volumes" Oct 30 22:08:43.316702 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.316677 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:08:43.396482 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396442 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396693 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396546 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396693 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396620 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396693 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396647 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396693 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396686 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396777 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:43.396918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396815 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.396918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396882 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:43.396918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396903 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.397139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396939 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.397139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.396972 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\") pod \"4b544a7d-7df3-4799-841c-f5069200e931\" (UID: \"4b544a7d-7df3-4799-841c-f5069200e931\") " Oct 30 22:08:43.397139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397057 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:43.397139 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397089 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:43.397365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397320 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:43.397586 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397537 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.397586 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397558 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b544a7d-7df3-4799-841c-f5069200e931-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.397586 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397573 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.397586 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397585 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.397823 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.397594 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.399226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.399196 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:43.399383 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.399359 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:43.399556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.399541 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:43.399597 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.399561 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws" pod "4b544a7d-7df3-4799-841c-f5069200e931" (UID: "4b544a7d-7df3-4799-841c-f5069200e931"). InnerVolumeSpecName "kube-api-access-fvtws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:43.498970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.498933 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtws\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-kube-api-access-fvtws\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.498970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.498965 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b544a7d-7df3-4799-841c-f5069200e931-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.498970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.498974 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b544a7d-7df3-4799-841c-f5069200e931-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.499191 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.498985 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b544a7d-7df3-4799-841c-f5069200e931-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:43.978509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.978475 4525 generic.go:356] "Generic (PLEG): container finished" podID="4b544a7d-7df3-4799-841c-f5069200e931" containerID="d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027" exitCode=0 Oct 30 22:08:43.978713 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.978526 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" Oct 30 22:08:43.978713 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.978557 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerDied","Data":"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027"} Oct 30 22:08:43.978713 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.978638 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf" event={"ID":"4b544a7d-7df3-4799-841c-f5069200e931","Type":"ContainerDied","Data":"5bbdb4640099f43545c36fe265e6824f7f477a8ab4044e18b4a782e02e3745d3"} Oct 30 22:08:43.978713 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.978669 4525 scope.go:122] "RemoveContainer" containerID="d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027" Oct 30 22:08:43.989081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.989061 4525 scope.go:122] "RemoveContainer" containerID="36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab" Oct 30 22:08:43.998403 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.998383 4525 scope.go:122] "RemoveContainer" containerID="d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027" Oct 30 22:08:43.998697 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:43.998680 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027\": container with ID starting with d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027 not found: ID does not exist" containerID="d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027" Oct 30 22:08:43.998756 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.998706 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027"} err="failed to get container status \"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027\": rpc error: code = NotFound desc = could not find container \"d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027\": container with ID starting with d28f844ef9dff981be78ffcef5157dd6429d3d0028ddc978ba8dcf334ec0d027 not found: ID does not exist" Oct 30 22:08:43.998756 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.998723 4525 scope.go:122] "RemoveContainer" containerID="36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab" Oct 30 22:08:43.998975 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:43.998951 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab\": container with ID starting with 36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab not found: ID does not exist" containerID="36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab" Oct 30 22:08:43.999018 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:43.998982 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab"} err="failed to get container status \"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab\": rpc error: code = NotFound desc = could not find container \"36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab\": container with ID starting with 36022615c7366420e89dda3320ce3848a28d105278a56500c6774ba06a3391ab not found: ID does not exist" Oct 30 22:08:44.156171 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:44.156130 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:08:44.205809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:44.205766 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ae1bd5-openshift-default-7dd5d8c9d9-cpgtf"] Oct 30 22:08:45.490749 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:45.490717 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b544a7d-7df3-4799-841c-f5069200e931" path="/var/lib/kubelet/pods/4b544a7d-7df3-4799-841c-f5069200e931/volumes" Oct 30 22:08:46.876506 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:46.876467 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 22:08:46.876988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:46.876726 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" containerID="cri-o://d87722582509fe0d7654b75fee04689e1f624820791c11892a211a08913de752" gracePeriod=30 Oct 30 22:08:48.902369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:48.902328 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:08:49.015134 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.015102 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:08:49.640412 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.640378 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 22:08:49.640641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.640575 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" podUID="25ad9478-4720-47a3-a9b6-3df6938e5315" containerName="maas-api" containerID="cri-o://b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb" gracePeriod=30 Oct 30 22:08:49.903125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.903063 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 22:08:49.925540 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.925518 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\" (UniqueName: \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\") pod \"25ad9478-4720-47a3-a9b6-3df6938e5315\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " Oct 30 22:08:49.925701 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.925684 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\") pod \"25ad9478-4720-47a3-a9b6-3df6938e5315\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " Oct 30 22:08:49.925779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.925727 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") pod \"25ad9478-4720-47a3-a9b6-3df6938e5315\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " Oct 30 22:08:49.925779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.925770 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\") pod \"25ad9478-4720-47a3-a9b6-3df6938e5315\" (UID: \"25ad9478-4720-47a3-a9b6-3df6938e5315\") " Oct 30 22:08:49.926080 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.926056 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca" pod "25ad9478-4720-47a3-a9b6-3df6938e5315" (UID: "25ad9478-4720-47a3-a9b6-3df6938e5315"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:49.926218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.926203 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ad9478-4720-47a3-a9b6-3df6938e5315-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:49.928228 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.928095 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc" pod "25ad9478-4720-47a3-a9b6-3df6938e5315" (UID: "25ad9478-4720-47a3-a9b6-3df6938e5315"). InnerVolumeSpecName "kube-api-access-wq7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:49.928228 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.928131 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls" pod "25ad9478-4720-47a3-a9b6-3df6938e5315" (UID: "25ad9478-4720-47a3-a9b6-3df6938e5315"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:49.928228 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:49.928147 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs" pod "25ad9478-4720-47a3-a9b6-3df6938e5315" (UID: "25ad9478-4720-47a3-a9b6-3df6938e5315"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:50.016407 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.016371 4525 generic.go:356] "Generic (PLEG): container finished" podID="25ad9478-4720-47a3-a9b6-3df6938e5315" containerID="b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb" exitCode=0 Oct 30 22:08:50.016593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.016415 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" Oct 30 22:08:50.016593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.016445 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" event={"ID":"25ad9478-4720-47a3-a9b6-3df6938e5315","Type":"ContainerDied","Data":"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb"} Oct 30 22:08:50.016593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.016539 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f" event={"ID":"25ad9478-4720-47a3-a9b6-3df6938e5315","Type":"ContainerDied","Data":"d6140c8ce9aee51860016592fb48f3ddabbe3003aced1098171e98f75df159a1"} Oct 30 22:08:50.016593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.016562 4525 scope.go:122] "RemoveContainer" containerID="b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb" Oct 30 22:08:50.028804 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.028776 4525 scope.go:122] "RemoveContainer" containerID="b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb" Oct 30 22:08:50.029462 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.029436 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq7xc\" (UniqueName: \"kubernetes.io/projected/25ad9478-4720-47a3-a9b6-3df6938e5315-kube-api-access-wq7xc\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:50.029550 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.029466 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:50.029550 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.029475 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ad9478-4720-47a3-a9b6-3df6938e5315-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:50.029962 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:50.029939 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb\": container with ID starting with b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb not found: ID does not exist" containerID="b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb" Oct 30 22:08:50.029962 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.029972 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb"} err="failed to get container status \"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb\": rpc error: code = NotFound desc = could not find container \"b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb\": container with ID starting with b2646a2f964ea805de970cc70f265b7ebd6b82dfa890bcf74d3cc5ca4de014eb not found: ID does not exist" Oct 30 22:08:50.218924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.218829 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 22:08:50.226046 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:50.226011 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-1a415f-858565cfc5-nz59f"] Oct 30 22:08:51.476210 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:51.476174 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ad9478-4720-47a3-a9b6-3df6938e5315" path="/var/lib/kubelet/pods/25ad9478-4720-47a3-a9b6-3df6938e5315/volumes" Oct 30 22:08:51.582519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:51.582486 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:08:51.660761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:51.660729 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:08:52.064315 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.064282 4525 generic.go:356] "Generic (PLEG): container finished" podID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerID="d87722582509fe0d7654b75fee04689e1f624820791c11892a211a08913de752" exitCode=0 Oct 30 22:08:52.064499 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.064353 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" event={"ID":"ccfb88d8-7d4e-436a-aaf5-79189fec1661","Type":"ContainerDied","Data":"d87722582509fe0d7654b75fee04689e1f624820791c11892a211a08913de752"} Oct 30 22:08:52.237726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.237693 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 22:08:52.286976 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.286942 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287014 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287116 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287184 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287262 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287305 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287346 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287393 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287444 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:52.287675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287534 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\") pod \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\" (UID: \"ccfb88d8-7d4e-436a-aaf5-79189fec1661\") " Oct 30 22:08:52.287675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287652 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:52.287992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.287930 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:52.288066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.288035 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:52.288066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.288048 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:52.288182 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.288161 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.288240 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.288190 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.288240 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.288211 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.289503 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.289475 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:52.290016 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.289990 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:52.290115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.290013 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:52.290115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.289995 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll" pod "ccfb88d8-7d4e-436a-aaf5-79189fec1661" (UID: "ccfb88d8-7d4e-436a-aaf5-79189fec1661"). InnerVolumeSpecName "kube-api-access-q4dll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:52.389373 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389333 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.389373 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389373 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.389570 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389388 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.389570 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389402 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.389570 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389416 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccfb88d8-7d4e-436a-aaf5-79189fec1661-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:52.389570 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:52.389430 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dll\" (UniqueName: \"kubernetes.io/projected/ccfb88d8-7d4e-436a-aaf5-79189fec1661-kube-api-access-q4dll\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.105672 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.105627 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" event={"ID":"ccfb88d8-7d4e-436a-aaf5-79189fec1661","Type":"ContainerDied","Data":"64f625e1fbff99d6f36e1bb757b09f3b84cfeb8b605b14eabe5b7037c9b90f4d"} Oct 30 22:08:53.106212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.105760 4525 scope.go:122] "RemoveContainer" containerID="d87722582509fe0d7654b75fee04689e1f624820791c11892a211a08913de752" Oct 30 22:08:53.106212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.106078 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs" Oct 30 22:08:53.280746 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.280700 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 22:08:53.345943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.340437 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-69e4b2-openshift-default-5f7bf6748f-dx2rs"] Oct 30 22:08:53.480697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.480644 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" path="/var/lib/kubelet/pods/ccfb88d8-7d4e-436a-aaf5-79189fec1661/volumes" Oct 30 22:08:53.902977 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:53.902921 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" podUID="bafe5ae7-f024-4175-8152-d870fec17fae" containerName="payload-processing" containerID="cri-o://752e3f9615b95862ecd0e37296b2ab80cc57312fa2a103538a3e1b176751c64e" gracePeriod=25 Oct 30 22:08:54.016012 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.015969 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" podUID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" containerName="payload-pre-processing" containerID="cri-o://1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2" gracePeriod=25 Oct 30 22:08:54.115939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.115905 4525 generic.go:356] "Generic (PLEG): container finished" podID="bafe5ae7-f024-4175-8152-d870fec17fae" containerID="752e3f9615b95862ecd0e37296b2ab80cc57312fa2a103538a3e1b176751c64e" exitCode=0 Oct 30 22:08:54.116426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.116081 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" event={"ID":"bafe5ae7-f024-4175-8152-d870fec17fae","Type":"ContainerDied","Data":"752e3f9615b95862ecd0e37296b2ab80cc57312fa2a103538a3e1b176751c64e"} Oct 30 22:08:54.169642 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.169531 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:08:54.224994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.224967 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\") pod \"bafe5ae7-f024-4175-8152-d870fec17fae\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " Oct 30 22:08:54.225175 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.225014 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\" (UniqueName: \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\") pod \"bafe5ae7-f024-4175-8152-d870fec17fae\" (UID: \"bafe5ae7-f024-4175-8152-d870fec17fae\") " Oct 30 22:08:54.225389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.225359 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume" pod "bafe5ae7-f024-4175-8152-d870fec17fae" (UID: "bafe5ae7-f024-4175-8152-d870fec17fae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:54.227770 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.227743 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779" pod "bafe5ae7-f024-4175-8152-d870fec17fae" (UID: "bafe5ae7-f024-4175-8152-d870fec17fae"). InnerVolumeSpecName "kube-api-access-p6779". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:54.279254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.279225 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:08:54.325981 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.325950 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\" (UniqueName: \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\") pod \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " Oct 30 22:08:54.326156 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.325990 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\") pod \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\" (UID: \"ec6d903d-ad05-4df9-ab60-7714a6c5583a\") " Oct 30 22:08:54.326297 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.326282 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bafe5ae7-f024-4175-8152-d870fec17fae-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:54.326369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.326299 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6779\" (UniqueName: \"kubernetes.io/projected/bafe5ae7-f024-4175-8152-d870fec17fae-kube-api-access-p6779\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:54.326524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.326498 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume" pod "ec6d903d-ad05-4df9-ab60-7714a6c5583a" (UID: "ec6d903d-ad05-4df9-ab60-7714a6c5583a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:54.328316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.328291 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7" pod "ec6d903d-ad05-4df9-ab60-7714a6c5583a" (UID: "ec6d903d-ad05-4df9-ab60-7714a6c5583a"). InnerVolumeSpecName "kube-api-access-bwzc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:54.427959 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.427881 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzc7\" (UniqueName: \"kubernetes.io/projected/ec6d903d-ad05-4df9-ab60-7714a6c5583a-kube-api-access-bwzc7\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:54.427959 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:54.427920 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6d903d-ad05-4df9-ab60-7714a6c5583a-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:55.124761 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.124729 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" Oct 30 22:08:55.125245 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.124757 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j" event={"ID":"bafe5ae7-f024-4175-8152-d870fec17fae","Type":"ContainerDied","Data":"ec5c1105ecb5a0baa10e8e254f4bb7643837ff82679644136ba18db9ac7dbbde"} Oct 30 22:08:55.125245 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.124826 4525 scope.go:122] "RemoveContainer" containerID="752e3f9615b95862ecd0e37296b2ab80cc57312fa2a103538a3e1b176751c64e" Oct 30 22:08:55.126592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.126425 4525 generic.go:356] "Generic (PLEG): container finished" podID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" containerID="1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2" exitCode=0 Oct 30 22:08:55.126592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.126475 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" Oct 30 22:08:55.126592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.126480 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" event={"ID":"ec6d903d-ad05-4df9-ab60-7714a6c5583a","Type":"ContainerDied","Data":"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2"} Oct 30 22:08:55.126592 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.126565 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb" event={"ID":"ec6d903d-ad05-4df9-ab60-7714a6c5583a","Type":"ContainerDied","Data":"1658ab3053127fb71823046c5355b4745ffd58c2af0f014ea71d04ffcc125089"} Oct 30 22:08:55.135122 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.135102 4525 scope.go:122] "RemoveContainer" containerID="1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2" Oct 30 22:08:55.144309 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.144291 4525 scope.go:122] "RemoveContainer" containerID="1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2" Oct 30 22:08:55.144627 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:55.144584 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2\": container with ID starting with 1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2 not found: ID does not exist" containerID="1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2" Oct 30 22:08:55.144691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.144636 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2"} err="failed to get container status \"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2\": rpc error: code = NotFound desc = could not find container \"1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2\": container with ID starting with 1fc9d7ad0eb840c8ed2f5bc5dc175cdcae58fb74b35ad5df42cf82c2551880c2 not found: ID does not exist" Oct 30 22:08:55.290433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.290348 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:08:55.341008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.340976 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5bf4f8-55c5b47cfd-bld9j"] Oct 30 22:08:55.408508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.408467 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:08:55.413319 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.413292 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5bf4f8-5cb4667b67-h6chb"] Oct 30 22:08:55.474021 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.473988 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafe5ae7-f024-4175-8152-d870fec17fae" path="/var/lib/kubelet/pods/bafe5ae7-f024-4175-8152-d870fec17fae/volumes" Oct 30 22:08:55.474404 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:55.474391 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" path="/var/lib/kubelet/pods/ec6d903d-ad05-4df9-ab60-7714a6c5583a/volumes" Oct 30 22:08:56.583733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.583696 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" podUID="839bddb2-d550-48b3-8043-a2beac8bd4b2" containerName="payload-processing" containerID="cri-o://ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9" gracePeriod=25 Oct 30 22:08:56.850888 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.850819 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:08:56.957299 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.957274 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\" (UniqueName: \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\") pod \"839bddb2-d550-48b3-8043-a2beac8bd4b2\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " Oct 30 22:08:56.957492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.957436 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\") pod \"839bddb2-d550-48b3-8043-a2beac8bd4b2\" (UID: \"839bddb2-d550-48b3-8043-a2beac8bd4b2\") " Oct 30 22:08:56.957995 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.957976 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume" pod "839bddb2-d550-48b3-8043-a2beac8bd4b2" (UID: "839bddb2-d550-48b3-8043-a2beac8bd4b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:56.959673 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:56.959583 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p" pod "839bddb2-d550-48b3-8043-a2beac8bd4b2" (UID: "839bddb2-d550-48b3-8043-a2beac8bd4b2"). InnerVolumeSpecName "kube-api-access-d797p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:57.059246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.059210 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839bddb2-d550-48b3-8043-a2beac8bd4b2-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.059246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.059238 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d797p\" (UniqueName: \"kubernetes.io/projected/839bddb2-d550-48b3-8043-a2beac8bd4b2-kube-api-access-d797p\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.065423 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.065388 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" podUID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" containerName="payload-pre-processing" containerID="cri-o://aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f" gracePeriod=25 Oct 30 22:08:57.148933 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.148892 4525 generic.go:356] "Generic (PLEG): container finished" podID="839bddb2-d550-48b3-8043-a2beac8bd4b2" containerID="ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9" exitCode=0 Oct 30 22:08:57.149104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.148935 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" event={"ID":"839bddb2-d550-48b3-8043-a2beac8bd4b2","Type":"ContainerDied","Data":"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9"} Oct 30 22:08:57.149104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.148956 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" Oct 30 22:08:57.149104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.149031 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5" event={"ID":"839bddb2-d550-48b3-8043-a2beac8bd4b2","Type":"ContainerDied","Data":"a0271d2c51c533bd401836543236be8bdf1e518f322efaa729505128a7e3ef07"} Oct 30 22:08:57.149104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.149066 4525 scope.go:122] "RemoveContainer" containerID="ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9" Oct 30 22:08:57.212486 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.212455 4525 scope.go:122] "RemoveContainer" containerID="ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9" Oct 30 22:08:57.212884 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:57.212858 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9\": container with ID starting with ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9 not found: ID does not exist" containerID="ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9" Oct 30 22:08:57.212984 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.212897 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9"} err="failed to get container status \"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9\": rpc error: code = NotFound desc = could not find container \"ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9\": container with ID starting with ed50b609734cfb243bb93568e66b4f76741cdcfa6ee7a780cfc57966b23e72d9 not found: ID does not exist" Oct 30 22:08:57.320725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.320686 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:08:57.322536 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.322511 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5a19c1-9bbbd8fb-zn2n5"] Oct 30 22:08:57.346308 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.346283 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:08:57.362818 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.362786 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\" (UniqueName: \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\") pod \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " Oct 30 22:08:57.362978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.362887 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\") pod \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\" (UID: \"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06\") " Oct 30 22:08:57.363293 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.363267 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume" pod "8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" (UID: "8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:57.363428 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.363412 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.365449 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.365420 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg" pod "8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" (UID: "8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06"). InnerVolumeSpecName "kube-api-access-8wvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:57.464666 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.464578 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvwg\" (UniqueName: \"kubernetes.io/projected/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06-kube-api-access-8wvwg\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:57.476392 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:57.476362 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839bddb2-d550-48b3-8043-a2beac8bd4b2" path="/var/lib/kubelet/pods/839bddb2-d550-48b3-8043-a2beac8bd4b2/volumes" Oct 30 22:08:58.156068 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.156028 4525 generic.go:356] "Generic (PLEG): container finished" podID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" containerID="aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f" exitCode=0 Oct 30 22:08:58.156579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.156082 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" Oct 30 22:08:58.156579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.156114 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" event={"ID":"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06","Type":"ContainerDied","Data":"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f"} Oct 30 22:08:58.156579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.156173 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs" event={"ID":"8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06","Type":"ContainerDied","Data":"8e33ef6dc9aecae772871175cd85a1eae341c34b41c7aac24c3efd3716b63510"} Oct 30 22:08:58.156579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.156206 4525 scope.go:122] "RemoveContainer" containerID="aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f" Oct 30 22:08:58.165822 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.165784 4525 scope.go:122] "RemoveContainer" containerID="aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f" Oct 30 22:08:58.166085 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:08:58.166061 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f\": container with ID starting with aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f not found: ID does not exist" containerID="aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f" Oct 30 22:08:58.166182 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.166096 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f"} err="failed to get container status \"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f\": rpc error: code = NotFound desc = could not find container \"aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f\": container with ID starting with aadd11901a090a4ecad9d6efb79001e599c983a4998e3ee695295d5158a5fd3f not found: ID does not exist" Oct 30 22:08:58.323326 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.323292 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:08:58.372557 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:58.372525 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5a19c1-5b6dfcbdf9-l52rs"] Oct 30 22:08:59.473920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:08:59.473887 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" path="/var/lib/kubelet/pods/8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06/volumes" Oct 30 22:09:05.396334 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:05.396292 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:09:08.982957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:08.982918 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 22:09:08.983492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:08.983191 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" containerID="cri-o://4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a" gracePeriod=30 Oct 30 22:09:11.314472 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:11.314429 4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:11.314882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:11.314498 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:14.250977 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.250889 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 22:09:14.253852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.253820 4525 generic.go:356] "Generic (PLEG): container finished" podID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerID="4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a" exitCode=0 Oct 30 22:09:14.254002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.253881 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" Oct 30 22:09:14.254002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.253884 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerDied","Data":"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a"} Oct 30 22:09:14.254002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.253941 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p" event={"ID":"f340ba07-fb78-4e39-8be7-c1fd6deac8ee","Type":"ContainerDied","Data":"d33191bb7573c310b0150708cfc74d8fc727f57a9987cabd85838590e362fb6f"} Oct 30 22:09:14.254002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.253970 4525 scope.go:122] "RemoveContainer" containerID="4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a" Oct 30 22:09:14.264172 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.264150 4525 scope.go:122] "RemoveContainer" containerID="f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0" Oct 30 22:09:14.273940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.273914 4525 scope.go:122] "RemoveContainer" containerID="4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a" Oct 30 22:09:14.274264 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:14.274244 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a\": container with ID starting with 4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a not found: ID does not exist" containerID="4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a" Oct 30 22:09:14.274363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.274278 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a"} err="failed to get container status \"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a\": rpc error: code = NotFound desc = could not find container \"4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a\": container with ID starting with 4722fda81dcb95d23472ae8214e1baaa99e5369fdfc2bbd626ea29cba1cf183a not found: ID does not exist" Oct 30 22:09:14.274363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.274306 4525 scope.go:122] "RemoveContainer" containerID="f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0" Oct 30 22:09:14.274600 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:14.274581 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0\": container with ID starting with f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0 not found: ID does not exist" containerID="f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0" Oct 30 22:09:14.274690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.274637 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0"} err="failed to get container status \"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0\": rpc error: code = NotFound desc = could not find container \"f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0\": container with ID starting with f5a9a8386499205446df1b66cfc3657f5eff074ebf2df9294058aa3bbd4312a0 not found: ID does not exist" Oct 30 22:09:14.376929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.376890 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.376940 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.376985 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377041 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377072 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377148 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377178 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377201 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:14.377352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377305 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377352 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377332 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:14.377707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377350 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\") pod \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\" (UID: \"f340ba07-fb78-4e39-8be7-c1fd6deac8ee\") " Oct 30 22:09:14.377707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377420 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:14.377707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377533 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:14.377707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.377666 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:14.378179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.378111 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.378238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.378185 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.378238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.378200 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.378238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.378214 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.378238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.378230 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.379525 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.379505 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:14.379668 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.379555 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "kube-api-access-7d5h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:14.379936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.379921 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:14.380010 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.379949 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy" pod "f340ba07-fb78-4e39-8be7-c1fd6deac8ee" (UID: "f340ba07-fb78-4e39-8be7-c1fd6deac8ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:14.479316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.479262 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5h9\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-kube-api-access-7d5h9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.479316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.479312 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.479316 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.479323 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.479558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.479333 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f340ba07-fb78-4e39-8be7-c1fd6deac8ee-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.657937 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.657901 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:09:14.682144 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.682110 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 22:09:14.688423 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:14.688394 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1a415f-openshift-default-76d8699997-c268p"] Oct 30 22:09:15.474148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:15.474117 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" path="/var/lib/kubelet/pods/f340ba07-fb78-4e39-8be7-c1fd6deac8ee/volumes" Oct 30 22:09:24.437507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:24.437465 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 22:09:31.794276 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:31.794239 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:09:31.794750 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:31.794439 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerName="maas-api" containerID="cri-o://1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01" gracePeriod=30 Oct 30 22:09:31.814800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:31.814761 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": dial tcp 10.133.0.162:8443: connect: connection refused" Oct 30 22:09:32.110956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.110918 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:09:32.206009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.205973 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") pod \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " Oct 30 22:09:32.206364 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.206338 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\") pod \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " Oct 30 22:09:32.206561 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.206544 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\") pod \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " Oct 30 22:09:32.206745 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.206727 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\") pod \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\" (UID: \"2b4b053e-7637-4d7b-b88c-1e8bc3a796da\") " Oct 30 22:09:32.219875 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.219815 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca" pod "2b4b053e-7637-4d7b-b88c-1e8bc3a796da" (UID: "2b4b053e-7637-4d7b-b88c-1e8bc3a796da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:32.244455 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.240509 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b" pod "2b4b053e-7637-4d7b-b88c-1e8bc3a796da" (UID: "2b4b053e-7637-4d7b-b88c-1e8bc3a796da"). InnerVolumeSpecName "kube-api-access-5qg8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:32.256800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.256753 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls" pod "2b4b053e-7637-4d7b-b88c-1e8bc3a796da" (UID: "2b4b053e-7637-4d7b-b88c-1e8bc3a796da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:32.257120 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.257080 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs" pod "2b4b053e-7637-4d7b-b88c-1e8bc3a796da" (UID: "2b4b053e-7637-4d7b-b88c-1e8bc3a796da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:32.308416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.308334 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:32.308416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.308365 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-kube-api-access-5qg8b\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:32.308416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.308378 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:32.308416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.308387 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b4b053e-7637-4d7b-b88c-1e8bc3a796da-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:32.386831 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.386794 4525 generic.go:356] "Generic (PLEG): container finished" podID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerID="1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01" exitCode=0 Oct 30 22:09:32.387119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.386858 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" Oct 30 22:09:32.387119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.386866 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" event={"ID":"2b4b053e-7637-4d7b-b88c-1e8bc3a796da","Type":"ContainerDied","Data":"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01"} Oct 30 22:09:32.387119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.386925 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc" event={"ID":"2b4b053e-7637-4d7b-b88c-1e8bc3a796da","Type":"ContainerDied","Data":"220794c7e20d2f6edb54d7f497d5c96d493013070db37e4d59ce538a257ea96d"} Oct 30 22:09:32.387119 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.386963 4525 scope.go:122] "RemoveContainer" containerID="1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01" Oct 30 22:09:32.397511 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.397492 4525 scope.go:122] "RemoveContainer" containerID="1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01" Oct 30 22:09:32.397846 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:32.397827 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01\": container with ID starting with 1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01 not found: ID does not exist" containerID="1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01" Oct 30 22:09:32.397911 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.397854 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01"} err="failed to get container status \"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01\": rpc error: code = NotFound desc = could not find container \"1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01\": container with ID starting with 1092fbdfa7c36fc0ab74b31bdb47bb65bf968d2dfbf2cf9944cc917ba6a9ab01 not found: ID does not exist" Oct 30 22:09:32.552123 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.552092 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:09:32.556503 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:32.556457 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5bf4f8-c68d568d5-h7lqc"] Oct 30 22:09:33.473478 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:33.473446 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" path="/var/lib/kubelet/pods/2b4b053e-7637-4d7b-b88c-1e8bc3a796da/volumes" Oct 30 22:09:35.396471 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:35.396432 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:09:39.458519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.458482 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" containerID="cri-o://5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669" gracePeriod=45 Oct 30 22:09:39.792517 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.792491 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 22:09:39.897509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897475 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.897509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897516 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.897843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897563 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.897843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897732 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.897843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897754 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.897843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897784 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.898062 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.898018 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.898062 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.898028 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.898062 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.897787 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\" (UniqueName: \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\") pod \"0156c42c-075a-4d86-96d5-758b85e0ea4c\" (UID: \"0156c42c-075a-4d86-96d5-758b85e0ea4c\") " Oct 30 22:09:39.898516 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.898498 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.898516 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.898516 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.898721 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.898525 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.900147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.900108 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:39.900491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.900433 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:39.900619 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.900583 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f" pod "0156c42c-075a-4d86-96d5-758b85e0ea4c" (UID: "0156c42c-075a-4d86-96d5-758b85e0ea4c"). InnerVolumeSpecName "kube-api-access-82q4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:39.999510 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.999479 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82q4f\" (UniqueName: \"kubernetes.io/projected/0156c42c-075a-4d86-96d5-758b85e0ea4c-kube-api-access-82q4f\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.999510 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.999507 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0156c42c-075a-4d86-96d5-758b85e0ea4c-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:39.999510 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:39.999517 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0156c42c-075a-4d86-96d5-758b85e0ea4c-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:40.432179 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.432144 4525 generic.go:356] "Generic (PLEG): container finished" podID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerID="5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669" exitCode=0 Oct 30 22:09:40.432398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.432190 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" event={"ID":"0156c42c-075a-4d86-96d5-758b85e0ea4c","Type":"ContainerDied","Data":"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669"} Oct 30 22:09:40.432398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.432249 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" event={"ID":"0156c42c-075a-4d86-96d5-758b85e0ea4c","Type":"ContainerDied","Data":"366fd9b3b8b1ff4a30f02666fbfc82cbf1777338831b4e48b89ac99bf2c8cf2a"} Oct 30 22:09:40.432398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.432273 4525 scope.go:122] "RemoveContainer" containerID="5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669" Oct 30 22:09:40.432398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.432201 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b" Oct 30 22:09:40.446050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.446030 4525 scope.go:122] "RemoveContainer" containerID="5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669" Oct 30 22:09:40.446367 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:09:40.446349 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669\": container with ID starting with 5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669 not found: ID does not exist" containerID="5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669" Oct 30 22:09:40.446414 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.446375 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669"} err="failed to get container status \"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669\": rpc error: code = NotFound desc = could not find container \"5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669\": container with ID starting with 5f18dd72b9aa4d0b4fb55d25dd933c65d3e42511d5ec435055defa61a21e2669 not found: ID does not exist" Oct 30 22:09:40.598450 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.598414 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 22:09:40.601175 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:40.601137 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-691598/distinct-2-w0-691598-kserve-85585b8488-grf8b"] Oct 30 22:09:41.547645 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:41.547584 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" path="/var/lib/kubelet/pods/0156c42c-075a-4d86-96d5-758b85e0ea4c/volumes" Oct 30 22:09:46.925397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:46.925358 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 22:09:46.975475 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:46.975440 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 22:09:46.977959 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:46.977930 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 22:09:54.000724 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:54.000688 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 22:09:59.001723 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:09:59.001676 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" containerID="cri-o://8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87" gracePeriod=25 Oct 30 22:10:01.949509 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:01.949438 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" containerID="cri-o://d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d" gracePeriod=45 Oct 30 22:10:01.996113 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:01.996050 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" containerID="cri-o://27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830" gracePeriod=45 Oct 30 22:10:01.997817 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:01.997760 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" containerID="cri-o://7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df" gracePeriod=45 Oct 30 22:10:02.030551 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.030514 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Oct 30 22:10:02.300081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.300052 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 22:10:02.339221 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.339196 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 22:10:02.342427 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.342407 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 22:10:02.382165 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382135 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.382165 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382173 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382227 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382272 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382317 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382347 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\" (UniqueName: \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382375 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.382433 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382425 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\" (UniqueName: \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382453 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382533 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382580 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382589 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382602 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382634 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382650 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382680 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\" (UniqueName: \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382712 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.382845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382769 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\") pod \"a44e0779-a66c-4beb-8147-63cecb20e3cb\" (UID: \"a44e0779-a66c-4beb-8147-63cecb20e3cb\") " Oct 30 22:10:02.383334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382864 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\") pod \"35c60783-4694-47d1-8e7e-c44a206867dd\" (UID: \"35c60783-4694-47d1-8e7e-c44a206867dd\") " Oct 30 22:10:02.383334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382906 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.383334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382908 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.383334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.382933 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\") pod \"3589c229-1f7b-481f-a040-0e14a166b7ee\" (UID: \"3589c229-1f7b-481f-a040-0e14a166b7ee\") " Oct 30 22:10:02.383334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383063 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.383655 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383404 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.383836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383811 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383823 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383865 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383884 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383900 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383915 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.383943 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.383929 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.384446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.384046 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.384446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.384149 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.388337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.388298 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.389000 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.388970 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "kube-api-access-sbpf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.389000 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.388978 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.389185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.388988 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "kube-api-access-f7xsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.389185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.389087 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs" pod "3589c229-1f7b-481f-a040-0e14a166b7ee" (UID: "3589c229-1f7b-481f-a040-0e14a166b7ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.389185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.389121 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:02.389365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.389224 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "kube-api-access-gbpdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:02.389798 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.389779 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs" pod "a44e0779-a66c-4beb-8147-63cecb20e3cb" (UID: "a44e0779-a66c-4beb-8147-63cecb20e3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.391071 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.391053 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs" pod "35c60783-4694-47d1-8e7e-c44a206867dd" (UID: "35c60783-4694-47d1-8e7e-c44a206867dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485136 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xsb\" (UniqueName: \"kubernetes.io/projected/3589c229-1f7b-481f-a040-0e14a166b7ee-kube-api-access-f7xsb\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485164 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485175 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485190 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpf6\" (UniqueName: \"kubernetes.io/projected/a44e0779-a66c-4beb-8147-63cecb20e3cb-kube-api-access-sbpf6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485204 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44e0779-a66c-4beb-8147-63cecb20e3cb-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485214 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44e0779-a66c-4beb-8147-63cecb20e3cb-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485223 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c60783-4694-47d1-8e7e-c44a206867dd-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485230 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485237 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3589c229-1f7b-481f-a040-0e14a166b7ee-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485244 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3589c229-1f7b-481f-a040-0e14a166b7ee-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485251 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpdm\" (UniqueName: \"kubernetes.io/projected/35c60783-4694-47d1-8e7e-c44a206867dd-kube-api-access-gbpdm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.485800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.485260 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c60783-4694-47d1-8e7e-c44a206867dd-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:02.599977 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.599945 4525 generic.go:356] "Generic (PLEG): container finished" podID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerID="7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df" exitCode=0 Oct 30 22:10:02.600190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.600012 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" Oct 30 22:10:02.600190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.600036 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" event={"ID":"3589c229-1f7b-481f-a040-0e14a166b7ee","Type":"ContainerDied","Data":"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df"} Oct 30 22:10:02.600190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.600085 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p" event={"ID":"3589c229-1f7b-481f-a040-0e14a166b7ee","Type":"ContainerDied","Data":"27905bc545ddce6de73010e45885c35ea959d320a10cf4dc808a2dd9b217e5f0"} Oct 30 22:10:02.600190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.600111 4525 scope.go:122] "RemoveContainer" containerID="7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df" Oct 30 22:10:02.602003 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.601967 4525 generic.go:356] "Generic (PLEG): container finished" podID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerID="d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d" exitCode=0 Oct 30 22:10:02.602151 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.602028 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" Oct 30 22:10:02.602151 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.602095 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" event={"ID":"a44e0779-a66c-4beb-8147-63cecb20e3cb","Type":"ContainerDied","Data":"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d"} Oct 30 22:10:02.602259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.602147 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp" event={"ID":"a44e0779-a66c-4beb-8147-63cecb20e3cb","Type":"ContainerDied","Data":"ec17520e297ecf3bad968b70cecea6904b88b14a0ef1f3f2d22899d457b2de97"} Oct 30 22:10:02.605083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.605061 4525 generic.go:356] "Generic (PLEG): container finished" podID="35c60783-4694-47d1-8e7e-c44a206867dd" containerID="27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830" exitCode=0 Oct 30 22:10:02.605212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.605110 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" Oct 30 22:10:02.605212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.605147 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" event={"ID":"35c60783-4694-47d1-8e7e-c44a206867dd","Type":"ContainerDied","Data":"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830"} Oct 30 22:10:02.605212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.605186 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl" event={"ID":"35c60783-4694-47d1-8e7e-c44a206867dd","Type":"ContainerDied","Data":"730b2bfb5ca7d026c6096fb99e31331f652d95a7abbca181d48231c2280a7ab8"} Oct 30 22:10:02.615961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.615931 4525 scope.go:122] "RemoveContainer" containerID="7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df" Oct 30 22:10:02.616516 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:02.616472 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df\": container with ID starting with 7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df not found: ID does not exist" containerID="7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df" Oct 30 22:10:02.616673 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.616549 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df"} err="failed to get container status \"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df\": rpc error: code = NotFound desc = could not find container \"7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df\": container with ID starting with 7c210132525e6f182a5df81d6ff1ccd60453dccf203fe0e658c2d11d71ba30df not found: ID does not exist" Oct 30 22:10:02.616673 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.616579 4525 scope.go:122] "RemoveContainer" containerID="d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d" Oct 30 22:10:02.629111 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.629088 4525 scope.go:122] "RemoveContainer" containerID="d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d" Oct 30 22:10:02.629422 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:02.629401 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d\": container with ID starting with d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d not found: ID does not exist" containerID="d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d" Oct 30 22:10:02.629507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.629436 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d"} err="failed to get container status \"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d\": rpc error: code = NotFound desc = could not find container \"d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d\": container with ID starting with d206f07c82316b255543d8a12b037d24fdf9ff88ae6210a4d6c511447842882d not found: ID does not exist" Oct 30 22:10:02.629507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.629464 4525 scope.go:122] "RemoveContainer" containerID="27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830" Oct 30 22:10:02.638531 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.638512 4525 scope.go:122] "RemoveContainer" containerID="27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830" Oct 30 22:10:02.638919 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:02.638897 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830\": container with ID starting with 27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830 not found: ID does not exist" containerID="27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830" Oct 30 22:10:02.639006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.638931 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830"} err="failed to get container status \"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830\": rpc error: code = NotFound desc = could not find container \"27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830\": container with ID starting with 27eef3e0bb72c4cfa3f50b4910c6799ab39f83400d0fb231929f469357e4b830 not found: ID does not exist" Oct 30 22:10:02.802398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.802303 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 22:10:02.807197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.807159 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-2-w4-1de75a-kserve-f9b766db5-p7kkp"] Oct 30 22:10:02.915444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.915358 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 22:10:02.918678 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:02.918656 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/premium-w4-1de75a-kserve-6598c87747-5jd4p"] Oct 30 22:10:03.028501 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:03.028460 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 22:10:03.037312 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:03.037277 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1de75a/distinct-w4-1de75a-kserve-55b7749f4-xjlvl"] Oct 30 22:10:03.474919 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:03.474887 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" path="/var/lib/kubelet/pods/3589c229-1f7b-481f-a040-0e14a166b7ee/volumes" Oct 30 22:10:03.475372 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:03.475358 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" path="/var/lib/kubelet/pods/35c60783-4694-47d1-8e7e-c44a206867dd/volumes" Oct 30 22:10:03.475764 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:03.475750 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" path="/var/lib/kubelet/pods/a44e0779-a66c-4beb-8147-63cecb20e3cb/volumes" Oct 30 22:10:04.627058 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:04.627026 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 22:10:05.395930 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:05.395896 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:10:07.378541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:07.378455 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:10:17.377811 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:17.377768 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:10:18.785898 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:18.785852 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:10:19.377884 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.377852 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 22:10:19.514101 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.514014 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\") pod \"4abb797a-663c-4e44-9903-f7797b2f6b55\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " Oct 30 22:10:19.514453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.514431 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\" (UniqueName: \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\") pod \"4abb797a-663c-4e44-9903-f7797b2f6b55\" (UID: \"4abb797a-663c-4e44-9903-f7797b2f6b55\") " Oct 30 22:10:19.519078 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.519047 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume" pod "4abb797a-663c-4e44-9903-f7797b2f6b55" (UID: "4abb797a-663c-4e44-9903-f7797b2f6b55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:19.522793 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.522768 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd" pod "4abb797a-663c-4e44-9903-f7797b2f6b55" (UID: "4abb797a-663c-4e44-9903-f7797b2f6b55"). InnerVolumeSpecName "kube-api-access-mtfhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.625554 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.625502 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfhd\" (UniqueName: \"kubernetes.io/projected/4abb797a-663c-4e44-9903-f7797b2f6b55-kube-api-access-mtfhd\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.625554 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.625549 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4abb797a-663c-4e44-9903-f7797b2f6b55-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.649437 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.649361 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" containerID="cri-o://7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875" gracePeriod=45 Oct 30 22:10:19.714167 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.714130 4525 generic.go:356] "Generic (PLEG): container finished" podID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerID="8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87" exitCode=0 Oct 30 22:10:19.714353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.714179 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" Oct 30 22:10:19.714353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.714206 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" event={"ID":"4abb797a-663c-4e44-9903-f7797b2f6b55","Type":"ContainerDied","Data":"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87"} Oct 30 22:10:19.714353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.714257 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd" event={"ID":"4abb797a-663c-4e44-9903-f7797b2f6b55","Type":"ContainerDied","Data":"736b02782915abfa35481cbf828fccbfb344907aed801ebfe90ebca5b44f3855"} Oct 30 22:10:19.714353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.714278 4525 scope.go:122] "RemoveContainer" containerID="8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87" Oct 30 22:10:19.763961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.763930 4525 scope.go:122] "RemoveContainer" containerID="8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87" Oct 30 22:10:19.764396 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:19.764327 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87\": container with ID starting with 8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87 not found: ID does not exist" containerID="8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87" Oct 30 22:10:19.764396 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.764372 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87"} err="failed to get container status \"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87\": rpc error: code = NotFound desc = could not find container \"8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87\": container with ID starting with 8eb55db04396e7791abead7fcc2e809d815dc00e66ba6a052ade59fc44010a87 not found: ID does not exist" Oct 30 22:10:19.890319 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.890242 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 22:10:19.916632 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.899255 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1de75a-b464f5df6-9whzd"] Oct 30 22:10:19.964009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:19.963978 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 22:10:20.054654 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054589 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.054807 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054691 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\" (UniqueName: \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.054807 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054719 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.054807 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054740 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.055005 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054863 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.055005 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054889 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\") pod \"27bd8c87-1929-45e7-a617-18015b60bcbd\" (UID: \"27bd8c87-1929-45e7-a617-18015b60bcbd\") " Oct 30 22:10:20.055005 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.054904 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.055137 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.055019 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.055137 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.055029 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.055313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.055294 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.055313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.055313 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.055442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.055321 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.057186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.057156 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:20.057312 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.057197 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:20.057312 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.057259 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh" pod "27bd8c87-1929-45e7-a617-18015b60bcbd" (UID: "27bd8c87-1929-45e7-a617-18015b60bcbd"). InnerVolumeSpecName "kube-api-access-7bdsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:20.156266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.156232 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27bd8c87-1929-45e7-a617-18015b60bcbd-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.156266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.156264 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27bd8c87-1929-45e7-a617-18015b60bcbd-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.156471 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.156274 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bdsh\" (UniqueName: \"kubernetes.io/projected/27bd8c87-1929-45e7-a617-18015b60bcbd-kube-api-access-7bdsh\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:20.721134 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.721104 4525 generic.go:356] "Generic (PLEG): container finished" podID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerID="7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875" exitCode=0 Oct 30 22:10:20.721337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.721148 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" Oct 30 22:10:20.721337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.721185 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" event={"ID":"27bd8c87-1929-45e7-a617-18015b60bcbd","Type":"ContainerDied","Data":"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875"} Oct 30 22:10:20.721337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.721228 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4" event={"ID":"27bd8c87-1929-45e7-a617-18015b60bcbd","Type":"ContainerDied","Data":"259d6ee067ed495e9df81028249e07f64969e4514abd89cfc1e7e55b5d133b33"} Oct 30 22:10:20.721337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.721248 4525 scope.go:122] "RemoveContainer" containerID="7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875" Oct 30 22:10:20.731402 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.731374 4525 scope.go:122] "RemoveContainer" containerID="7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875" Oct 30 22:10:20.731792 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:20.731768 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875\": container with ID starting with 7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875 not found: ID does not exist" containerID="7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875" Oct 30 22:10:20.731879 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.731808 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875"} err="failed to get container status \"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875\": rpc error: code = NotFound desc = could not find container \"7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875\": container with ID starting with 7c5ba081c6aec251c86d3ca48abc77fe96157db83346b2624745c9db27058875 not found: ID does not exist" Oct 30 22:10:20.888927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.888897 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 22:10:20.938873 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:20.938839 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6ea374/premium-w1-6ea374-kserve-f88b69587-78wb4"] Oct 30 22:10:21.476678 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:21.476646 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" path="/var/lib/kubelet/pods/27bd8c87-1929-45e7-a617-18015b60bcbd/volumes" Oct 30 22:10:21.477051 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:21.477037 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" path="/var/lib/kubelet/pods/4abb797a-663c-4e44-9903-f7797b2f6b55/volumes" Oct 30 22:10:35.396031 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:35.395990 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.073485 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074522 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074539 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074569 4525 cpu_manager.go:414] "containerMap: removing container" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074578 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074592 4525 cpu_manager.go:414] "containerMap: removing container" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074597 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074622 4525 cpu_manager.go:414] "containerMap: removing container" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074630 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074641 4525 cpu_manager.go:414] "containerMap: removing container" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074647 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074658 4525 cpu_manager.go:414] "containerMap: removing container" podUID="839bddb2-d550-48b3-8043-a2beac8bd4b2" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074665 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839bddb2-d550-48b3-8043-a2beac8bd4b2" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074676 4525 cpu_manager.go:414] "containerMap: removing container" podUID="bafe5ae7-f024-4175-8152-d870fec17fae" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074681 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafe5ae7-f024-4175-8152-d870fec17fae" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074688 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074692 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074698 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074705 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074711 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074715 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074733 4525 cpu_manager.go:414] "containerMap: removing container" podUID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074741 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074755 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074762 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074778 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074783 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074789 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074793 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074807 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074815 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074825 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074832 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074841 4525 cpu_manager.go:414] "containerMap: removing container" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074846 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074855 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074859 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074866 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074870 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074887 4525 cpu_manager.go:414] "containerMap: removing container" podUID="25ad9478-4720-47a3-a9b6-3df6938e5315" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074894 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ad9478-4720-47a3-a9b6-3df6938e5315" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074989 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8034a8ec-b2fd-4d29-8d8b-71b2bb59ba06" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.074999 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44e0779-a66c-4beb-8147-63cecb20e3cb" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075010 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075015 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4b053e-7637-4d7b-b88c-1e8bc3a796da" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075022 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075027 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abb797a-663c-4e44-9903-f7797b2f6b55" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075033 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075039 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6d903d-ad05-4df9-ab60-7714a6c5583a" containerName="payload-pre-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075046 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03d13fc-4001-4c40-bfe7-ff6402a14ab2" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075058 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ad9478-4720-47a3-a9b6-3df6938e5315" containerName="maas-api" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075069 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075079 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3589c229-1f7b-481f-a040-0e14a166b7ee" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075086 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5e7f17-3426-4d62-af1a-181e70840c70" containerName="istio-proxy" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075092 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="27bd8c87-1929-45e7-a617-18015b60bcbd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075097 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c60783-4694-47d1-8e7e-c44a206867dd" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075101 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="839bddb2-d550-48b3-8043-a2beac8bd4b2" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075107 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafe5ae7-f024-4175-8152-d870fec17fae" containerName="payload-processing" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075113 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0156c42c-075a-4d86-96d5-758b85e0ea4c" containerName="main" Oct 30 22:10:48.078633 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.075118 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfb88d8-7d4e-436a-aaf5-79189fec1661" containerName="istio-proxy" Oct 30 22:10:48.088044 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.088015 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.148718 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.148688 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6ea374-openshift-default-dockercfg-jjxj8\"" Oct 30 22:10:48.199520 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.199488 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:10:48.203453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203425 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203465 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvvs\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203505 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203621 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203643 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203665 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203681 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203753 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.203900 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.203815 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.304832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304798 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.304832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304839 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvvs\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304870 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304925 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304947 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304979 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.304998 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305053 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305021 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305364 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305073 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305364 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305280 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305364 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305353 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305496 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305439 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305496 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305461 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.305928 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.305911 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.307440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.307418 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.307618 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.307591 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.367114 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.367081 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.367623 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.367586 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvvs\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\") pod \"e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.404130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.404102 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:48.859932 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.859905 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:10:48.862293 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:10:48.862262 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0721afed_862b_4402_8f71_0dea67fd2bbb.slice/crio-eaf4e7b3175cd1dd859ff7b1aac319ca9c1863d0a1cddc9fcb5cccb756585713 WatchSource:0}: Error finding container eaf4e7b3175cd1dd859ff7b1aac319ca9c1863d0a1cddc9fcb5cccb756585713: Status 404 returned error can't find the container with id eaf4e7b3175cd1dd859ff7b1aac319ca9c1863d0a1cddc9fcb5cccb756585713 Oct 30 22:10:48.864728 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.864697 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:48.864819 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.864763 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:48.864819 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.864797 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:48.875156 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:48.875130 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" event={"ID":"0721afed-862b-4402-8f71-0dea67fd2bbb","Type":"ContainerStarted","Data":"eaf4e7b3175cd1dd859ff7b1aac319ca9c1863d0a1cddc9fcb5cccb756585713"} Oct 30 22:10:49.881919 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:49.881881 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" event={"ID":"0721afed-862b-4402-8f71-0dea67fd2bbb","Type":"ContainerStarted","Data":"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8"} Oct 30 22:10:50.107812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:50.107757 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podStartSLOduration=2.107740688 podStartE2EDuration="2.107740688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.055141722 +0000 UTC m=+2874.729188257" watchObservedRunningTime="2026-10-30 22:10:50.107740688 +0000 UTC m=+2874.781787223" Oct 30 22:10:50.405209 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:50.405173 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:50.406692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:50.406658 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:50.406809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:50.406717 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:51.405086 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:51.405054 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:51.405524 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:51.405108 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:52.405041 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:52.405001 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:52.405225 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:52.405059 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:53.405189 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.405146 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:53.405477 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.405210 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:53.459831 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.459794 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 22:10:53.460016 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.459982 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" podUID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" containerName="maas-api" containerID="cri-o://2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43" gracePeriod=30 Oct 30 22:10:53.799131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.799104 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 22:10:53.909653 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.909585 4525 generic.go:356] "Generic (PLEG): container finished" podID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" containerID="2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43" exitCode=0 Oct 30 22:10:53.909834 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.909689 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" Oct 30 22:10:53.909834 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.909700 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" event={"ID":"abe516fb-955b-4a93-8dd8-63c2bd851b8c","Type":"ContainerDied","Data":"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43"} Oct 30 22:10:53.909834 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.909752 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854" event={"ID":"abe516fb-955b-4a93-8dd8-63c2bd851b8c","Type":"ContainerDied","Data":"f0b5db6ac91e2a827f1a05eedf78efb67381f64ec7621b39f9a11c93e0e3b594"} Oct 30 22:10:53.909834 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.909803 4525 scope.go:122] "RemoveContainer" containerID="2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43" Oct 30 22:10:53.920295 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.920272 4525 scope.go:122] "RemoveContainer" containerID="2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43" Oct 30 22:10:53.920598 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:10:53.920577 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43\": container with ID starting with 2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43 not found: ID does not exist" containerID="2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43" Oct 30 22:10:53.920795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.920624 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43"} err="failed to get container status \"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43\": rpc error: code = NotFound desc = could not find container \"2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43\": container with ID starting with 2a46ec44807b72d8ffae9ef3bfcefb3d2c73e63b437d492f4151504e2e069c43 not found: ID does not exist" Oct 30 22:10:53.944118 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.943013 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") pod \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " Oct 30 22:10:53.944118 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.943113 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\" (UniqueName: \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\") pod \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " Oct 30 22:10:53.944118 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.943155 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\") pod \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " Oct 30 22:10:53.944118 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.943189 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\") pod \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\" (UID: \"abe516fb-955b-4a93-8dd8-63c2bd851b8c\") " Oct 30 22:10:53.945466 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.945441 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca" pod "abe516fb-955b-4a93-8dd8-63c2bd851b8c" (UID: "abe516fb-955b-4a93-8dd8-63c2bd851b8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:53.955957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.955903 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls" pod "abe516fb-955b-4a93-8dd8-63c2bd851b8c" (UID: "abe516fb-955b-4a93-8dd8-63c2bd851b8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:53.956782 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.956741 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs" pod "abe516fb-955b-4a93-8dd8-63c2bd851b8c" (UID: "abe516fb-955b-4a93-8dd8-63c2bd851b8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:53.960631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:53.960012 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6" pod "abe516fb-955b-4a93-8dd8-63c2bd851b8c" (UID: "abe516fb-955b-4a93-8dd8-63c2bd851b8c"). InnerVolumeSpecName "kube-api-access-qsbh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:54.047685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.046456 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.047685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.046499 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsbh6\" (UniqueName: \"kubernetes.io/projected/abe516fb-955b-4a93-8dd8-63c2bd851b8c-kube-api-access-qsbh6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.047685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.046514 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe516fb-955b-4a93-8dd8-63c2bd851b8c-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.047685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.046532 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abe516fb-955b-4a93-8dd8-63c2bd851b8c-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:54.348690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.344017 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 22:10:54.382471 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.382435 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd2a94-696dbf78c8-dc854"] Oct 30 22:10:54.404927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.404878 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:54.405127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.404948 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:54.759717 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.759678 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 22:10:54.760150 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.759967 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" podUID="30798ac3-f1bc-40bc-bc83-5914da439947" containerName="maas-api" containerID="cri-o://243851cc76e2f3edba64c15bb5b0dd4299de38f61706c68f827617d906537321" gracePeriod=30 Oct 30 22:10:54.980133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.979062 4525 generic.go:356] "Generic (PLEG): container finished" podID="30798ac3-f1bc-40bc-bc83-5914da439947" containerID="243851cc76e2f3edba64c15bb5b0dd4299de38f61706c68f827617d906537321" exitCode=0 Oct 30 22:10:54.980133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:54.979217 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" event={"ID":"30798ac3-f1bc-40bc-bc83-5914da439947","Type":"ContainerDied","Data":"243851cc76e2f3edba64c15bb5b0dd4299de38f61706c68f827617d906537321"} Oct 30 22:10:55.101339 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.101297 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 22:10:55.167219 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167185 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\" (UniqueName: \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\") pod \"30798ac3-f1bc-40bc-bc83-5914da439947\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " Oct 30 22:10:55.167419 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167273 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\") pod \"30798ac3-f1bc-40bc-bc83-5914da439947\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " Oct 30 22:10:55.167419 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167328 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\") pod \"30798ac3-f1bc-40bc-bc83-5914da439947\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " Oct 30 22:10:55.167553 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167482 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") pod \"30798ac3-f1bc-40bc-bc83-5914da439947\" (UID: \"30798ac3-f1bc-40bc-bc83-5914da439947\") " Oct 30 22:10:55.167708 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167683 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca" pod "30798ac3-f1bc-40bc-bc83-5914da439947" (UID: "30798ac3-f1bc-40bc-bc83-5914da439947"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:55.168020 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.167993 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30798ac3-f1bc-40bc-bc83-5914da439947-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.169682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.169578 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2" pod "30798ac3-f1bc-40bc-bc83-5914da439947" (UID: "30798ac3-f1bc-40bc-bc83-5914da439947"). InnerVolumeSpecName "kube-api-access-rkzj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:55.169682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.169588 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs" pod "30798ac3-f1bc-40bc-bc83-5914da439947" (UID: "30798ac3-f1bc-40bc-bc83-5914da439947"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:55.169682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.169650 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls" pod "30798ac3-f1bc-40bc-bc83-5914da439947" (UID: "30798ac3-f1bc-40bc-bc83-5914da439947"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:55.269720 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.269685 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.269720 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.269720 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30798ac3-f1bc-40bc-bc83-5914da439947-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.270013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.269737 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzj2\" (UniqueName: \"kubernetes.io/projected/30798ac3-f1bc-40bc-bc83-5914da439947-kube-api-access-rkzj2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:55.405006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.404976 4525 patch_prober.go:29] interesting pod/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 22:10:55.405188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.405028 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 22:10:55.473465 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.473429 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" path="/var/lib/kubelet/pods/abe516fb-955b-4a93-8dd8-63c2bd851b8c/volumes" Oct 30 22:10:55.994066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.994035 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" Oct 30 22:10:55.994066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.994045 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7" event={"ID":"30798ac3-f1bc-40bc-bc83-5914da439947","Type":"ContainerDied","Data":"05d927bd4afad3b16cd01a557ef0d2d61d7453836850a7668ea4b3900930f0ed"} Oct 30 22:10:55.994708 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:55.994105 4525 scope.go:122] "RemoveContainer" containerID="243851cc76e2f3edba64c15bb5b0dd4299de38f61706c68f827617d906537321" Oct 30 22:10:56.160908 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:56.160866 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 22:10:56.210843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:56.210796 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1de75a-8c99f9998-nx8g7"] Oct 30 22:10:56.631743 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:56.631711 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:56.632194 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:56.632163 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:56.633319 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:56.633298 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:10:57.473155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:57.473126 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30798ac3-f1bc-40bc-bc83-5914da439947" path="/var/lib/kubelet/pods/30798ac3-f1bc-40bc-bc83-5914da439947/volumes" Oct 30 22:10:58.548220 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:58.548187 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 22:10:58.548692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:58.548457 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" containerID="cri-o://d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172" gracePeriod=30 Oct 30 22:10:58.597988 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:58.597949 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 22:10:58.598224 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:10:58.598200 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" containerID="cri-o://2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5" gracePeriod=30 Oct 30 22:11:03.299947 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.299913 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:11:03.300369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.300116 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" containerID="cri-o://62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b" gracePeriod=30 Oct 30 22:11:03.403697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.399757 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mrk7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:03.403697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.399832 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:03.518914 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.518881 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xrbg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:03.519082 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.518940 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:03.887515 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.887485 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 22:11:03.905631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905572 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.905631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905618 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.905832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905643 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.905832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905710 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.905832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905781 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.905832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905810 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.906049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905870 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.906049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905892 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:03.906049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.905906 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.906049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906036 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\") pod \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\" (UID: \"4ea6b3ee-cbbf-44af-826e-647792e2cb7c\") " Oct 30 22:11:03.906257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906114 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:03.906417 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906395 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:03.906473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906411 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:03.906473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906415 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:03.906582 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906536 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.906582 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906554 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.906582 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906568 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.906785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906581 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.906785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.906596 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.908685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.908656 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "kube-api-access-zzrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:03.908820 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.908682 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:03.908918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.908897 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:03.908918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.908906 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo" pod "4ea6b3ee-cbbf-44af-826e-647792e2cb7c" (UID: "4ea6b3ee-cbbf-44af-826e-647792e2cb7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:03.913035 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:03.913017 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 22:11:04.007573 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007539 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.007573 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007579 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.007866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007637 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.007866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007661 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.007866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007698 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.007866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007728 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.008093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007883 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.008093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007926 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.008093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.007954 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.008093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008024 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.008093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008024 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.008374 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008133 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\") pod \"e39db119-794c-469d-a941-8f28110bd3bd\" (UID: \"e39db119-794c-469d-a941-8f28110bd3bd\") " Oct 30 22:11:04.008374 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008354 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.008500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008452 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008709 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008735 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrb2\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-kube-api-access-zzrb2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008751 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008767 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008780 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008794 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008809 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008822 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39db119-794c-469d-a941-8f28110bd3bd-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.008783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.008837 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea6b3ee-cbbf-44af-826e-647792e2cb7c-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.010226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.010200 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.010390 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.010370 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.010692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.010675 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "kube-api-access-9b6dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.010994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.010976 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo" pod "e39db119-794c-469d-a941-8f28110bd3bd" (UID: "e39db119-794c-469d-a941-8f28110bd3bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:04.077153 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.077113 4525 generic.go:356] "Generic (PLEG): container finished" podID="e39db119-794c-469d-a941-8f28110bd3bd" containerID="2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5" exitCode=0 Oct 30 22:11:04.077340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.077174 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" Oct 30 22:11:04.077340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.077195 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" event={"ID":"e39db119-794c-469d-a941-8f28110bd3bd","Type":"ContainerDied","Data":"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5"} Oct 30 22:11:04.077340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.077282 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8" event={"ID":"e39db119-794c-469d-a941-8f28110bd3bd","Type":"ContainerDied","Data":"bde517620206a9239be777867b4c5b9717d9075b52b84d2753f3d4acd204da60"} Oct 30 22:11:04.077340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.077312 4525 scope.go:122] "RemoveContainer" containerID="2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5" Oct 30 22:11:04.079812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.079761 4525 generic.go:356] "Generic (PLEG): container finished" podID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerID="d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172" exitCode=0 Oct 30 22:11:04.080018 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.079834 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" Oct 30 22:11:04.080250 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.079832 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" event={"ID":"4ea6b3ee-cbbf-44af-826e-647792e2cb7c","Type":"ContainerDied","Data":"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172"} Oct 30 22:11:04.080455 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.080429 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h" event={"ID":"4ea6b3ee-cbbf-44af-826e-647792e2cb7c","Type":"ContainerDied","Data":"276f97cf55261c8904c169a0d4667ac0c1d7788697af005679f3be2fe055175e"} Oct 30 22:11:04.091227 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.091199 4525 scope.go:122] "RemoveContainer" containerID="2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5" Oct 30 22:11:04.093806 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:04.093778 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5\": container with ID starting with 2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5 not found: ID does not exist" containerID="2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5" Oct 30 22:11:04.093936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.093815 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5"} err="failed to get container status \"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5\": rpc error: code = NotFound desc = could not find container \"2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5\": container with ID starting with 2e72d15c88587a9c616910c57cd156caf27bda7f780521c59a0af5337a0744a5 not found: ID does not exist" Oct 30 22:11:04.093936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.093835 4525 scope.go:122] "RemoveContainer" containerID="d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172" Oct 30 22:11:04.103411 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.103393 4525 scope.go:122] "RemoveContainer" containerID="d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172" Oct 30 22:11:04.103754 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:04.103726 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172\": container with ID starting with d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172 not found: ID does not exist" containerID="d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172" Oct 30 22:11:04.103866 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.103764 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172"} err="failed to get container status \"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172\": rpc error: code = NotFound desc = could not find container \"d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172\": container with ID starting with d90d803a297b390eacac80f4863e531f3e3b92252bc0201889d2f9fa89f95172 not found: ID does not exist" Oct 30 22:11:04.109677 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.109649 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39db119-794c-469d-a941-8f28110bd3bd-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.109774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.109683 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39db119-794c-469d-a941-8f28110bd3bd-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.109774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.109726 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b6dz\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-kube-api-access-9b6dz\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.109774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.109742 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39db119-794c-469d-a941-8f28110bd3bd-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.251676 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.251641 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 22:11:04.261320 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.261289 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xrbg8"] Oct 30 22:11:04.381580 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.381544 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 22:11:04.386327 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:04.386300 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mrk7h"] Oct 30 22:11:05.396025 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:05.395991 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:11:05.473223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:05.473188 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" path="/var/lib/kubelet/pods/4ea6b3ee-cbbf-44af-826e-647792e2cb7c/volumes" Oct 30 22:11:05.473742 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:05.473728 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39db119-794c-469d-a941-8f28110bd3bd" path="/var/lib/kubelet/pods/e39db119-794c-469d-a941-8f28110bd3bd/volumes" Oct 30 22:11:08.566890 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.566822 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:11:08.672443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672410 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672485 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672554 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672595 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672697 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672654 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672700 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672772 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672841 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.672916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672865 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.672916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.672882 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\") pod \"ee4f334c-f3fd-4052-926f-c10c16bd7049\" (UID: \"ee4f334c-f3fd-4052-926f-c10c16bd7049\") " Oct 30 22:11:08.673232 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673049 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.673232 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673132 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.673232 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673189 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:08.673405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673291 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.673405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673360 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.673405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673383 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee4f334c-f3fd-4052-926f-c10c16bd7049-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.673522 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673404 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.673522 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.673423 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.675230 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.675199 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:08.675349 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.675210 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:08.675349 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.675314 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "kube-api-access-8lq42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:08.675349 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.675339 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo" pod "ee4f334c-f3fd-4052-926f-c10c16bd7049" (UID: "ee4f334c-f3fd-4052-926f-c10c16bd7049"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:08.774939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.774910 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.774939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.774936 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.774939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.774947 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lq42\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-kube-api-access-8lq42\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.775176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.774958 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:08.775176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:08.774966 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee4f334c-f3fd-4052-926f-c10c16bd7049-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:09.114587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.114545 4525 generic.go:356] "Generic (PLEG): container finished" podID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerID="62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b" exitCode=0 Oct 30 22:11:09.114769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.114601 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" Oct 30 22:11:09.114769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.114641 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" event={"ID":"ee4f334c-f3fd-4052-926f-c10c16bd7049","Type":"ContainerDied","Data":"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b"} Oct 30 22:11:09.114769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.114700 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp" event={"ID":"ee4f334c-f3fd-4052-926f-c10c16bd7049","Type":"ContainerDied","Data":"90c48d8f2f3e86627242e95fdff09018403e2e5a9d6550b5ef64d192fbe238f7"} Oct 30 22:11:09.114769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.114721 4525 scope.go:122] "RemoveContainer" containerID="62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b" Oct 30 22:11:09.125199 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.125181 4525 scope.go:122] "RemoveContainer" containerID="62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b" Oct 30 22:11:09.125452 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:09.125432 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b\": container with ID starting with 62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b not found: ID does not exist" containerID="62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b" Oct 30 22:11:09.125495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.125462 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b"} err="failed to get container status \"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b\": rpc error: code = NotFound desc = could not find container \"62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b\": container with ID starting with 62de933a45c159c39fd0e28facf92edbecf8b0aa89dcfb9899fd23efaa95c02b not found: ID does not exist" Oct 30 22:11:09.286991 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.283649 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:11:09.332356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.332321 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd2a94-openshift-default-7bd4bf7c5f-7d2kp"] Oct 30 22:11:09.474580 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:09.474539 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" path="/var/lib/kubelet/pods/ee4f334c-f3fd-4052-926f-c10c16bd7049/volumes" Oct 30 22:11:16.037463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:16.037429 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:11:16.461891 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:16.461844 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:11:16.463008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:16.462965 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" containerID="cri-o://139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8" gracePeriod=30 Oct 30 22:11:21.038208 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.038130 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" containerID="cri-o://fe7b8927218d2697c72f3f54ee6420c141094e4d20675557a597a8905819f9b6" gracePeriod=25 Oct 30 22:11:21.714670 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.714640 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:11:21.835104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835021 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835061 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835138 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835160 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835205 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835226 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835243 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835312 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835350 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835345 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\") pod \"0721afed-862b-4402-8f71-0dea67fd2bbb\" (UID: \"0721afed-862b-4402-8f71-0dea67fd2bbb\") " Oct 30 22:11:21.835774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835443 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.835774 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835630 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.835916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835781 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.835916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835800 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.835916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835824 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:21.835916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835905 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.835916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.835940 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.837571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.837543 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "kube-api-access-9kvvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:21.837763 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.837742 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:21.837935 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.837920 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:21.838019 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.837949 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo" pod "0721afed-862b-4402-8f71-0dea67fd2bbb" (UID: "0721afed-862b-4402-8f71-0dea67fd2bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:21.936946 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.936909 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvvs\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-kube-api-access-9kvvs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.936946 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.936952 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.937159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.936974 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.937159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.936984 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.937159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.936992 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.937159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.937000 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0721afed-862b-4402-8f71-0dea67fd2bbb-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.937159 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:21.937008 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0721afed-862b-4402-8f71-0dea67fd2bbb-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:22.190396 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.190354 4525 generic.go:356] "Generic (PLEG): container finished" podID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerID="139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8" exitCode=0 Oct 30 22:11:22.190872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.190392 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" event={"ID":"0721afed-862b-4402-8f71-0dea67fd2bbb","Type":"ContainerDied","Data":"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8"} Oct 30 22:11:22.190872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.190429 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" Oct 30 22:11:22.190872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.190458 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr" event={"ID":"0721afed-862b-4402-8f71-0dea67fd2bbb","Type":"ContainerDied","Data":"eaf4e7b3175cd1dd859ff7b1aac319ca9c1863d0a1cddc9fcb5cccb756585713"} Oct 30 22:11:22.190872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.190482 4525 scope.go:122] "RemoveContainer" containerID="139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8" Oct 30 22:11:22.200554 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.200535 4525 scope.go:122] "RemoveContainer" containerID="139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8" Oct 30 22:11:22.200847 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:22.200827 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8\": container with ID starting with 139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8 not found: ID does not exist" containerID="139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8" Oct 30 22:11:22.200924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.200853 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8"} err="failed to get container status \"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8\": rpc error: code = NotFound desc = could not find container \"139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8\": container with ID starting with 139a7b9fb6e38edde84484ec1db9c0d3ec770c65b5a43fa88dc7b7810c3997a8 not found: ID does not exist" Oct 30 22:11:22.356581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.356549 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:11:22.360939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:22.360913 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6ea374-openshift-default-84cc8c4467-vvrhr"] Oct 30 22:11:23.446480 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:23.446443 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:11:23.473686 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:23.473656 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" path="/var/lib/kubelet/pods/0721afed-862b-4402-8f71-0dea67fd2bbb/volumes" Oct 30 22:11:23.763999 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:23.763918 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:11:26.838272 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:26.838239 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:11:33.114872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.114839 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:11:33.115389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.115096 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" podUID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" containerName="maas-api" containerID="cri-o://2676062877e0cac05b580e2fee9df68fd505c090d1b5af1ef1f434f61cc0c078" gracePeriod=30 Oct 30 22:11:33.252781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.252740 4525 generic.go:356] "Generic (PLEG): container finished" podID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" containerID="2676062877e0cac05b580e2fee9df68fd505c090d1b5af1ef1f434f61cc0c078" exitCode=0 Oct 30 22:11:33.252936 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.252814 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" event={"ID":"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e","Type":"ContainerDied","Data":"2676062877e0cac05b580e2fee9df68fd505c090d1b5af1ef1f434f61cc0c078"} Oct 30 22:11:33.377861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.377836 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:11:33.447021 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.446986 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:11:33.449239 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449217 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\" (UniqueName: \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\") pod \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " Oct 30 22:11:33.449356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449341 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") pod \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " Oct 30 22:11:33.449400 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449389 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\") pod \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " Oct 30 22:11:33.449458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449426 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\") pod \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\" (UID: \"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e\") " Oct 30 22:11:33.449754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449728 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca" pod "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" (UID: "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:33.449904 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.449887 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:33.451497 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.451474 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g" pod "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" (UID: "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e"). InnerVolumeSpecName "kube-api-access-ptt6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:33.451644 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.451563 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls" pod "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" (UID: "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:33.451644 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.451577 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs" pod "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" (UID: "a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:33.550624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.550572 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:33.550624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.550619 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:33.550624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:33.550631 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptt6g\" (UniqueName: \"kubernetes.io/projected/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e-kube-api-access-ptt6g\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:34.259299 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:34.259262 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" Oct 30 22:11:34.259806 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:34.259292 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb" event={"ID":"a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e","Type":"ContainerDied","Data":"0679902e41a9a124a4aa29a9f34362ab330a6729836b8a459c9937f52399e12c"} Oct 30 22:11:34.259806 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:34.259358 4525 scope.go:122] "RemoveContainer" containerID="2676062877e0cac05b580e2fee9df68fd505c090d1b5af1ef1f434f61cc0c078" Oct 30 22:11:34.425830 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:34.425791 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:11:34.475872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:34.475838 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea1f0b-8497b7ff68-9sspb"] Oct 30 22:11:35.396727 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:35.396691 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:11:35.472783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:35.472753 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" path="/var/lib/kubelet/pods/a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e/volumes" Oct 30 22:11:41.297896 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.297813 4525 generic.go:356] "Generic (PLEG): container finished" podID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerID="fe7b8927218d2697c72f3f54ee6420c141094e4d20675557a597a8905819f9b6" exitCode=0 Oct 30 22:11:41.297896 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.297880 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" event={"ID":"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5","Type":"ContainerDied","Data":"fe7b8927218d2697c72f3f54ee6420c141094e4d20675557a597a8905819f9b6"} Oct 30 22:11:41.298388 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.297932 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" event={"ID":"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5","Type":"ContainerDied","Data":"f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f"} Oct 30 22:11:41.298388 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.297948 4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00bece314ea37eb082a84446b64b8c4ceec0a10e9c6e31172f9731aab9fce6f" Oct 30 22:11:41.302964 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.302942 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:11:41.337836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.337800 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\" (UniqueName: \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\") pod \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " Oct 30 22:11:41.338015 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.337865 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\") pod \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\" (UID: \"a9ee4b7e-8af1-4e33-9811-a71f24cb49d5\") " Oct 30 22:11:41.338361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.338335 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume" pod "a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" (UID: "a9ee4b7e-8af1-4e33-9811-a71f24cb49d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:41.340163 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.340134 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf" pod "a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" (UID: "a9ee4b7e-8af1-4e33-9811-a71f24cb49d5"). InnerVolumeSpecName "kube-api-access-45bwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:41.439953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.439916 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bwf\" (UniqueName: \"kubernetes.io/projected/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-kube-api-access-45bwf\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:41.439953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.439956 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:41.858564 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:41.858524 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" podUID="3c2f241c-8757-4636-a978-34ef53274d57" containerName="main" containerID="cri-o://fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06" gracePeriod=45 Oct 30 22:11:42.107952 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.107928 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:11:42.145010 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.144979 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145024 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145076 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\" (UniqueName: \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145140 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145200 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145259 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\") pod \"3c2f241c-8757-4636-a978-34ef53274d57\" (UID: \"3c2f241c-8757-4636-a978-34ef53274d57\") " Oct 30 22:11:42.145376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145306 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:42.145376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145335 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:42.145588 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145462 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:42.145709 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145689 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.145764 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145714 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.145764 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.145727 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.147711 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.147684 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:42.147818 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.147687 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:42.147818 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.147750 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g" pod "3c2f241c-8757-4636-a978-34ef53274d57" (UID: "3c2f241c-8757-4636-a978-34ef53274d57"). InnerVolumeSpecName "kube-api-access-blj5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:42.246270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.246240 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c2f241c-8757-4636-a978-34ef53274d57-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.246270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.246268 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c2f241c-8757-4636-a978-34ef53274d57-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.246479 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.246280 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj5g\" (UniqueName: \"kubernetes.io/projected/3c2f241c-8757-4636-a978-34ef53274d57-kube-api-access-blj5g\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:42.305457 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305421 4525 generic.go:356] "Generic (PLEG): container finished" podID="3c2f241c-8757-4636-a978-34ef53274d57" containerID="fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06" exitCode=0 Oct 30 22:11:42.305931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305466 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" Oct 30 22:11:42.305931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305501 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" event={"ID":"3c2f241c-8757-4636-a978-34ef53274d57","Type":"ContainerDied","Data":"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06"} Oct 30 22:11:42.305931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305542 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd" event={"ID":"3c2f241c-8757-4636-a978-34ef53274d57","Type":"ContainerDied","Data":"725536e2a73f6b37c4d72bbf78de9bf50b2c6d48506c115910c14f6b39629e16"} Oct 30 22:11:42.305931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305559 4525 scope.go:122] "RemoveContainer" containerID="fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06" Oct 30 22:11:42.305931 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.305753 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97" Oct 30 22:11:42.314713 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.314658 4525 scope.go:122] "RemoveContainer" containerID="fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06" Oct 30 22:11:42.314926 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:42.314903 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06\": container with ID starting with fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06 not found: ID does not exist" containerID="fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06" Oct 30 22:11:42.314990 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.314942 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06"} err="failed to get container status \"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06\": rpc error: code = NotFound desc = could not find container \"fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06\": container with ID starting with fb88d348baa386c83c40b8aa5c697edf1c2cbc60a6f39fe1c1f73d65c924fa06 not found: ID does not exist" Oct 30 22:11:42.491979 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.491887 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:11:42.502812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.502779 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ea1f0b-67fcfcdb5f-8dx97"] Oct 30 22:11:42.579464 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.579425 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:11:42.579687 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.579653 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" containerID="cri-o://706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf" gracePeriod=30 Oct 30 22:11:42.647863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.647833 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:11:42.681748 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:42.681708 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e6dcc0/test-model-e6dcc0-kserve-99b56cdb9-46fnd"] Oct 30 22:11:43.472383 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:43.472341 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2f241c-8757-4636-a978-34ef53274d57" path="/var/lib/kubelet/pods/3c2f241c-8757-4636-a978-34ef53274d57/volumes" Oct 30 22:11:43.472957 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:43.472930 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" path="/var/lib/kubelet/pods/a9ee4b7e-8af1-4e33-9811-a71f24cb49d5/volumes" Oct 30 22:11:47.835258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.835233 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:11:47.904089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904057 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904102 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904143 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904177 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904208 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904246 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904317 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904349 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904366 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\") pod \"5327b3bf-027c-43b3-a35e-3ed832f71182\" (UID: \"5327b3bf-027c-43b3-a35e-3ed832f71182\") " Oct 30 22:11:47.904652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904397 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:47.904879 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904696 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:47.904879 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904703 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:47.904986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904882 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:47.904986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904905 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5327b3bf-027c-43b3-a35e-3ed832f71182-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:47.904986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904921 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:47.904986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904877 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:47.904986 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.904885 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:47.906599 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.906576 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:47.907090 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.907070 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:47.907152 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.907121 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:47.907152 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:47.907129 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4" pod "5327b3bf-027c-43b3-a35e-3ed832f71182" (UID: "5327b3bf-027c-43b3-a35e-3ed832f71182"). InnerVolumeSpecName "kube-api-access-mv2m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:48.005680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005632 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.005680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005662 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.005680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005671 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.005680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005680 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.005680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005688 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5327b3bf-027c-43b3-a35e-3ed832f71182-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.006181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.005697 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2m4\" (UniqueName: \"kubernetes.io/projected/5327b3bf-027c-43b3-a35e-3ed832f71182-kube-api-access-mv2m4\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:48.338389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.338304 4525 generic.go:356] "Generic (PLEG): container finished" podID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerID="706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf" exitCode=0 Oct 30 22:11:48.338389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.338356 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" Oct 30 22:11:48.338622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.338386 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerDied","Data":"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf"} Oct 30 22:11:48.338622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.338441 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns" event={"ID":"5327b3bf-027c-43b3-a35e-3ed832f71182","Type":"ContainerDied","Data":"836646ac05fb8d81fdb9a21c38ffedbd2af30454423016f8265874e3e72d10eb"} Oct 30 22:11:48.338622 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.338466 4525 scope.go:122] "RemoveContainer" containerID="706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf" Oct 30 22:11:48.348824 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.348808 4525 scope.go:122] "RemoveContainer" containerID="7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87" Oct 30 22:11:48.357096 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.357077 4525 scope.go:122] "RemoveContainer" containerID="706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf" Oct 30 22:11:48.357371 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:48.357344 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf\": container with ID starting with 706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf not found: ID does not exist" containerID="706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf" Oct 30 22:11:48.357463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.357385 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf"} err="failed to get container status \"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf\": rpc error: code = NotFound desc = could not find container \"706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf\": container with ID starting with 706d02d00f5deedf5a2fac4a93556dbd4940398b17dc4341564fcc08659bbfcf not found: ID does not exist" Oct 30 22:11:48.357463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.357408 4525 scope.go:122] "RemoveContainer" containerID="7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87" Oct 30 22:11:48.357736 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:11:48.357717 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87\": container with ID starting with 7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87 not found: ID does not exist" containerID="7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87" Oct 30 22:11:48.357795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.357747 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87"} err="failed to get container status \"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87\": rpc error: code = NotFound desc = could not find container \"7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87\": container with ID starting with 7970df3a065d42b950d7d24e947f8ff594125e99002ef2c5311e96fc47d4cb87 not found: ID does not exist" Oct 30 22:11:48.504037 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.504008 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:11:48.554081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:48.554046 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ea1f0b-openshift-default-68fdcdb78c-l79ns"] Oct 30 22:11:49.472679 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:49.472631 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" path="/var/lib/kubelet/pods/5327b3bf-027c-43b3-a35e-3ed832f71182/volumes" Oct 30 22:11:57.066727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:11:57.060952 4525 scope.go:122] "RemoveContainer" containerID="fe7b8927218d2697c72f3f54ee6420c141094e4d20675557a597a8905819f9b6" Oct 30 22:12:01.785416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:01.785378 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:12:05.396642 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:12:05.396591 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:12:10.977710 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:10.977675 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:12:11.134460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:11.134425 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:12:15.978816 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:15.978771 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" containerID="cri-o://ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44" gracePeriod=25 Oct 30 22:12:16.135589 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:16.135528 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" containerID="cri-o://c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3" gracePeriod=25 Oct 30 22:12:21.369701 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.369663 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 22:12:21.370123 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.369880 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" podUID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" containerName="maas-api" containerID="cri-o://1eb56e8ee69be5e37b15a72d3afb9533946cb62b4a212d84936b031062fc25d8" gracePeriod=30 Oct 30 22:12:21.502115 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.502086 4525 generic.go:356] "Generic (PLEG): container finished" podID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" containerID="1eb56e8ee69be5e37b15a72d3afb9533946cb62b4a212d84936b031062fc25d8" exitCode=0 Oct 30 22:12:21.502279 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.502167 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" event={"ID":"c9e1b468-709d-4e9e-b975-2d732b6bdc40","Type":"ContainerDied","Data":"1eb56e8ee69be5e37b15a72d3afb9533946cb62b4a212d84936b031062fc25d8"} Oct 30 22:12:21.617478 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.617453 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 22:12:21.737216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.737135 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") pod \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " Oct 30 22:12:21.737392 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.737241 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\" (UniqueName: \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\") pod \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " Oct 30 22:12:21.737392 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.737274 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\") pod \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " Oct 30 22:12:21.737392 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.737331 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\") pod \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\" (UID: \"c9e1b468-709d-4e9e-b975-2d732b6bdc40\") " Oct 30 22:12:21.737729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.737703 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca" pod "c9e1b468-709d-4e9e-b975-2d732b6bdc40" (UID: "c9e1b468-709d-4e9e-b975-2d732b6bdc40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:21.739946 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.739917 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls" pod "c9e1b468-709d-4e9e-b975-2d732b6bdc40" (UID: "c9e1b468-709d-4e9e-b975-2d732b6bdc40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:21.740409 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.740367 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs" pod "c9e1b468-709d-4e9e-b975-2d732b6bdc40" (UID: "c9e1b468-709d-4e9e-b975-2d732b6bdc40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:21.740523 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.740450 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg" pod "c9e1b468-709d-4e9e-b975-2d732b6bdc40" (UID: "c9e1b468-709d-4e9e-b975-2d732b6bdc40"). InnerVolumeSpecName "kube-api-access-5z2xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:21.838541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.838507 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:21.838541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.838536 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9e1b468-709d-4e9e-b975-2d732b6bdc40-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:21.838541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.838546 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z2xg\" (UniqueName: \"kubernetes.io/projected/c9e1b468-709d-4e9e-b975-2d732b6bdc40-kube-api-access-5z2xg\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:21.838813 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:21.838557 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e1b468-709d-4e9e-b975-2d732b6bdc40-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:22.045141 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.045059 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:22.509229 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.509195 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" event={"ID":"c9e1b468-709d-4e9e-b975-2d732b6bdc40","Type":"ContainerDied","Data":"e072773c0d327ed241253cf8a3bff2e5758f1a01a0804dbd47c607a5dcb00e25"} Oct 30 22:12:22.509701 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.509255 4525 scope.go:122] "RemoveContainer" containerID="1eb56e8ee69be5e37b15a72d3afb9533946cb62b4a212d84936b031062fc25d8" Oct 30 22:12:22.509701 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.509210 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd" Oct 30 22:12:22.681925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.681893 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 22:12:22.731128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:22.731094 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-c7d477-789fb7fb76-m8cqd"] Oct 30 22:12:23.053619 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:23.053563 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:23.474338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:23.474299 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" path="/var/lib/kubelet/pods/c9e1b468-709d-4e9e-b975-2d732b6bdc40/volumes" Oct 30 22:12:30.364200 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:30.364162 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:12:32.045186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:32.045150 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:33.053234 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:33.053197 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:12:35.395828 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:12:35.395788 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:12:36.261164 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.261139 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:12:36.383277 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.383247 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:12:36.391238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.391212 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\" (UniqueName: \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\") pod \"987f3950-569e-47e4-96b7-995d0289b09a\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " Oct 30 22:12:36.391405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.391247 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\") pod \"987f3950-569e-47e4-96b7-995d0289b09a\" (UID: \"987f3950-569e-47e4-96b7-995d0289b09a\") " Oct 30 22:12:36.391711 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.391686 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume" pod "987f3950-569e-47e4-96b7-995d0289b09a" (UID: "987f3950-569e-47e4-96b7-995d0289b09a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:36.393452 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.393430 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2" pod "987f3950-569e-47e4-96b7-995d0289b09a" (UID: "987f3950-569e-47e4-96b7-995d0289b09a"). InnerVolumeSpecName "kube-api-access-n62m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:36.492252 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.492163 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\") pod \"508779a4-73f5-48e8-869d-4e54f50fef0a\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " Oct 30 22:12:36.492252 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.492215 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\" (UniqueName: \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\") pod \"508779a4-73f5-48e8-869d-4e54f50fef0a\" (UID: \"508779a4-73f5-48e8-869d-4e54f50fef0a\") " Oct 30 22:12:36.492797 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.492632 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62m2\" (UniqueName: \"kubernetes.io/projected/987f3950-569e-47e4-96b7-995d0289b09a-kube-api-access-n62m2\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.492797 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.492639 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume" pod "508779a4-73f5-48e8-869d-4e54f50fef0a" (UID: "508779a4-73f5-48e8-869d-4e54f50fef0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:36.492797 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.492659 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987f3950-569e-47e4-96b7-995d0289b09a-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.494526 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.494509 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm" pod "508779a4-73f5-48e8-869d-4e54f50fef0a" (UID: "508779a4-73f5-48e8-869d-4e54f50fef0a"). InnerVolumeSpecName "kube-api-access-qbghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:36.583280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.583247 4525 generic.go:356] "Generic (PLEG): container finished" podID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerID="c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3" exitCode=0 Oct 30 22:12:36.583492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.583292 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" Oct 30 22:12:36.583492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.583329 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" event={"ID":"508779a4-73f5-48e8-869d-4e54f50fef0a","Type":"ContainerDied","Data":"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3"} Oct 30 22:12:36.583492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.583387 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f" event={"ID":"508779a4-73f5-48e8-869d-4e54f50fef0a","Type":"ContainerDied","Data":"8880154d33a82bd79334087a657bcebb2c6a3d08e046015a8bf7e4414f38c7e7"} Oct 30 22:12:36.583492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.583414 4525 scope.go:122] "RemoveContainer" containerID="c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3" Oct 30 22:12:36.584709 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.584690 4525 generic.go:356] "Generic (PLEG): container finished" podID="987f3950-569e-47e4-96b7-995d0289b09a" containerID="ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44" exitCode=0 Oct 30 22:12:36.584838 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.584736 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" event={"ID":"987f3950-569e-47e4-96b7-995d0289b09a","Type":"ContainerDied","Data":"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44"} Oct 30 22:12:36.584838 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.584747 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" Oct 30 22:12:36.584838 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.584770 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h" event={"ID":"987f3950-569e-47e4-96b7-995d0289b09a","Type":"ContainerDied","Data":"f0bb7fa4c160942a473bd770f9dfef29691ac7cbacc8470fdbb530106034a74f"} Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.593245 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbghm\" (UniqueName: \"kubernetes.io/projected/508779a4-73f5-48e8-869d-4e54f50fef0a-kube-api-access-qbghm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.593272 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508779a4-73f5-48e8-869d-4e54f50fef0a-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.593754 4525 scope.go:122] "RemoveContainer" containerID="c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3" Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:12:36.594122 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3\": container with ID starting with c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3 not found: ID does not exist" containerID="c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3" Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.594154 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3"} err="failed to get container status \"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3\": rpc error: code = NotFound desc = could not find container \"c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3\": container with ID starting with c6f8cb9ee2fdb8ae6730dbd649c6573df54a4cbfd35a1d4343e599cf840e8dd3 not found: ID does not exist" Oct 30 22:12:36.594692 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.594181 4525 scope.go:122] "RemoveContainer" containerID="ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44" Oct 30 22:12:36.604949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.604929 4525 scope.go:122] "RemoveContainer" containerID="ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44" Oct 30 22:12:36.605251 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:12:36.605222 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44\": container with ID starting with ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44 not found: ID does not exist" containerID="ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44" Oct 30 22:12:36.605305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.605261 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44"} err="failed to get container status \"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44\": rpc error: code = NotFound desc = could not find container \"ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44\": container with ID starting with ba599d805d6688149e4f6bd243181cb2938b570477bc5a8ef1969cd18030bb44 not found: ID does not exist" Oct 30 22:12:36.750676 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.750581 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:12:36.757999 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.757970 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e6dcc0-66c8b6644c-8j98f"] Oct 30 22:12:36.883546 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.883512 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:12:36.888542 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:36.888518 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e6dcc0-96565f546-b9b5h"] Oct 30 22:12:37.473244 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:37.473211 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" path="/var/lib/kubelet/pods/508779a4-73f5-48e8-869d-4e54f50fef0a/volumes" Oct 30 22:12:37.473731 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:37.473708 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987f3950-569e-47e4-96b7-995d0289b09a" path="/var/lib/kubelet/pods/987f3950-569e-47e4-96b7-995d0289b09a/volumes" Oct 30 22:12:37.484617 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:37.484580 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:12:37.484878 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:37.484833 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" containerID="cri-o://20e385ec3fffdcceffb3ba4942c4c9b5996acfa5ae03752cff09a647ef68a04f" gracePeriod=30 Oct 30 22:12:42.623872 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.623835 4525 generic.go:356] "Generic (PLEG): container finished" podID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerID="20e385ec3fffdcceffb3ba4942c4c9b5996acfa5ae03752cff09a647ef68a04f" exitCode=0 Oct 30 22:12:42.624261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.623910 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerDied","Data":"20e385ec3fffdcceffb3ba4942c4c9b5996acfa5ae03752cff09a647ef68a04f"} Oct 30 22:12:42.624261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.623987 4525 scope.go:122] "RemoveContainer" containerID="6b33408d899402aaa07623e8f87042ae60ff056cdb5c3c407862efd27bf47849" Oct 30 22:12:42.741071 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.741041 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:12:42.851759 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851673 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.851759 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851723 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852004 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851765 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852004 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851869 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852004 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851962 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852004 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.851998 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852030 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852039 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852061 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852077 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852122 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:42.852226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852121 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\") pod \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\" (UID: \"86c1417d-e609-42c9-a9f7-afde6ed45e5c\") " Oct 30 22:12:42.852542 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852266 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:42.852542 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852394 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:42.852719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852541 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.852719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852567 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.852719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852584 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.852719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852596 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.852719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.852633 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.854376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.854351 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:42.854489 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.854382 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "kube-api-access-8pmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:42.854489 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.854434 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:42.854489 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.854445 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token" pod "86c1417d-e609-42c9-a9f7-afde6ed45e5c" (UID: "86c1417d-e609-42c9-a9f7-afde6ed45e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:42.954072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.954042 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.954072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.954071 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmwk\" (UniqueName: \"kubernetes.io/projected/86c1417d-e609-42c9-a9f7-afde6ed45e5c-kube-api-access-8pmwk\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.954072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.954080 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:42.954308 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:42.954090 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c1417d-e609-42c9-a9f7-afde6ed45e5c-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:43.631008 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:43.630966 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" event={"ID":"86c1417d-e609-42c9-a9f7-afde6ed45e5c","Type":"ContainerDied","Data":"e099ad11708b9d57f59bbf1383317b15ce599f7297bdbf8d67e359088d83c21a"} Oct 30 22:12:43.631496 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:43.631016 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb" Oct 30 22:12:43.631496 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:43.631037 4525 scope.go:122] "RemoveContainer" containerID="20e385ec3fffdcceffb3ba4942c4c9b5996acfa5ae03752cff09a647ef68a04f" Oct 30 22:12:43.798087 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:43.798003 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:12:43.848006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:43.847969 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e6dcc0-openshift-default-698cd5c4d5-7zdbb"] Oct 30 22:12:45.472714 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:45.472673 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" path="/var/lib/kubelet/pods/86c1417d-e609-42c9-a9f7-afde6ed45e5c/volumes" Oct 30 22:12:50.781379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:12:50.781342 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:13:00.016170 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:00.016122 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:13:00.199156 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:00.199121 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:13:05.017450 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.017407 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" podUID="1672e201-f9de-436b-a775-f0617c9dbfbd" containerName="payload-processing" containerID="cri-o://3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5" gracePeriod=25 Oct 30 22:13:05.200357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.200268 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" containerID="cri-o://9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97" gracePeriod=25 Oct 30 22:13:05.265365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.265336 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:13:05.354036 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.353957 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\") pod \"1672e201-f9de-436b-a775-f0617c9dbfbd\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " Oct 30 22:13:05.354036 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.354026 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\" (UniqueName: \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\") pod \"1672e201-f9de-436b-a775-f0617c9dbfbd\" (UID: \"1672e201-f9de-436b-a775-f0617c9dbfbd\") " Oct 30 22:13:05.354413 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.354381 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume" pod "1672e201-f9de-436b-a775-f0617c9dbfbd" (UID: "1672e201-f9de-436b-a775-f0617c9dbfbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:05.356268 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.356245 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj" pod "1672e201-f9de-436b-a775-f0617c9dbfbd" (UID: "1672e201-f9de-436b-a775-f0617c9dbfbd"). InnerVolumeSpecName "kube-api-access-4jzpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:05.397898 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:13:05.397860 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:13:05.455069 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.455033 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzpj\" (UniqueName: \"kubernetes.io/projected/1672e201-f9de-436b-a775-f0617c9dbfbd-kube-api-access-4jzpj\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:05.455069 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.455064 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1672e201-f9de-436b-a775-f0617c9dbfbd-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:05.734485 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.734447 4525 generic.go:356] "Generic (PLEG): container finished" podID="1672e201-f9de-436b-a775-f0617c9dbfbd" containerID="3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5" exitCode=0 Oct 30 22:13:05.734704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.734493 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" Oct 30 22:13:05.734704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.734493 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" event={"ID":"1672e201-f9de-436b-a775-f0617c9dbfbd","Type":"ContainerDied","Data":"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5"} Oct 30 22:13:05.734704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.734648 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw" event={"ID":"1672e201-f9de-436b-a775-f0617c9dbfbd","Type":"ContainerDied","Data":"83614783da3f5a09f03fbbfbb340d577b3afad82a71a1b7a7c9614222b99aa80"} Oct 30 22:13:05.734704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.734682 4525 scope.go:122] "RemoveContainer" containerID="3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5" Oct 30 22:13:05.743769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.743745 4525 scope.go:122] "RemoveContainer" containerID="3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5" Oct 30 22:13:05.744030 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:13:05.744011 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5\": container with ID starting with 3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5 not found: ID does not exist" containerID="3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5" Oct 30 22:13:05.744082 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.744038 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5"} err="failed to get container status \"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5\": rpc error: code = NotFound desc = could not find container \"3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5\": container with ID starting with 3ae63e79959d075d86e90441a6949719b4ab13d0d72f7fdb7adaa280006d95a5 not found: ID does not exist" Oct 30 22:13:05.906491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.906457 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:13:05.909083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:05.909058 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19bc92-865c656db-dn9kw"] Oct 30 22:13:07.475735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:07.475644 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1672e201-f9de-436b-a775-f0617c9dbfbd" path="/var/lib/kubelet/pods/1672e201-f9de-436b-a775-f0617c9dbfbd/volumes" Oct 30 22:13:08.056876 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:08.056839 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:13:12.804266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:12.804224 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:17.564725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:17.564686 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 22:13:17.614285 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:17.614252 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 22:13:22.804457 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:22.804417 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:13:25.455441 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.455414 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:13:25.552483 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.552389 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\") pod \"46d205b6-099b-47c6-a8b3-c6e18380473f\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " Oct 30 22:13:25.552483 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.552472 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\" (UniqueName: \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\") pod \"46d205b6-099b-47c6-a8b3-c6e18380473f\" (UID: \"46d205b6-099b-47c6-a8b3-c6e18380473f\") " Oct 30 22:13:25.552887 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.552863 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume" pod "46d205b6-099b-47c6-a8b3-c6e18380473f" (UID: "46d205b6-099b-47c6-a8b3-c6e18380473f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:25.554829 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.554809 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm" pod "46d205b6-099b-47c6-a8b3-c6e18380473f" (UID: "46d205b6-099b-47c6-a8b3-c6e18380473f"). InnerVolumeSpecName "kube-api-access-xtwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:25.654079 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.654046 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d205b6-099b-47c6-a8b3-c6e18380473f-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:25.654079 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.654077 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwzm\" (UniqueName: \"kubernetes.io/projected/46d205b6-099b-47c6-a8b3-c6e18380473f-kube-api-access-xtwzm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:25.827980 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.827898 4525 generic.go:356] "Generic (PLEG): container finished" podID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerID="9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97" exitCode=0 Oct 30 22:13:25.827980 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.827948 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" Oct 30 22:13:25.828178 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.827985 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" event={"ID":"46d205b6-099b-47c6-a8b3-c6e18380473f","Type":"ContainerDied","Data":"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97"} Oct 30 22:13:25.828178 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.828043 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg" event={"ID":"46d205b6-099b-47c6-a8b3-c6e18380473f","Type":"ContainerDied","Data":"9a7aaefaf9bedb713414906ae1d16b211ff909bab73be4f3a8e73e60bddd717c"} Oct 30 22:13:25.828178 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.828067 4525 scope.go:122] "RemoveContainer" containerID="9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97" Oct 30 22:13:25.837282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.837260 4525 scope.go:122] "RemoveContainer" containerID="9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97" Oct 30 22:13:25.837530 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:13:25.837510 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97\": container with ID starting with 9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97 not found: ID does not exist" containerID="9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97" Oct 30 22:13:25.837625 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.837539 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97"} err="failed to get container status \"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97\": rpc error: code = NotFound desc = could not find container \"9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97\": container with ID starting with 9b3a14361c9d0377ad46d8219d9e00b46a8ddb41e754cf102f7c12b1a1526e97 not found: ID does not exist" Oct 30 22:13:25.992562 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.992525 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:13:25.998209 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:25.998176 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19bc92-5d86488755-bj6mg"] Oct 30 22:13:27.472457 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:27.472424 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" path="/var/lib/kubelet/pods/46d205b6-099b-47c6-a8b3-c6e18380473f/volumes" Oct 30 22:13:32.584742 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.584705 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" containerID="cri-o://4a0067a264ebfc0732de5ad83d142dae1b3233702f39c44f20e5902fba82e3de" gracePeriod=45 Oct 30 22:13:32.634143 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.634110 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" containerID="cri-o://1f53bd9f71ac5d98ba60e7f3c56f5f6dbad7840db85d0bcfd92789c6e13977b7" gracePeriod=45 Oct 30 22:13:32.862236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.862199 4525 generic.go:356] "Generic (PLEG): container finished" podID="eaaed7af-6507-493a-867d-880a4c5f1395" containerID="4a0067a264ebfc0732de5ad83d142dae1b3233702f39c44f20e5902fba82e3de" exitCode=0 Oct 30 22:13:32.862432 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.862276 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" event={"ID":"eaaed7af-6507-493a-867d-880a4c5f1395","Type":"ContainerDied","Data":"4a0067a264ebfc0732de5ad83d142dae1b3233702f39c44f20e5902fba82e3de"} Oct 30 22:13:32.864354 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.864320 4525 generic.go:356] "Generic (PLEG): container finished" podID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerID="1f53bd9f71ac5d98ba60e7f3c56f5f6dbad7840db85d0bcfd92789c6e13977b7" exitCode=0 Oct 30 22:13:32.864546 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.864387 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" event={"ID":"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5","Type":"ContainerDied","Data":"1f53bd9f71ac5d98ba60e7f3c56f5f6dbad7840db85d0bcfd92789c6e13977b7"} Oct 30 22:13:32.884874 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.884850 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 22:13:32.915779 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.915756 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 22:13:32.927402 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927380 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.927495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927408 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927438 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.927495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927461 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.927495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927488 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927512 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927536 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927559 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927584 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\" (UniqueName: \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927646 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\") pod \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\" (UID: \"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927678 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\" (UniqueName: \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.927729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927694 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.928034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927706 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\") pod \"eaaed7af-6507-493a-867d-880a4c5f1395\" (UID: \"eaaed7af-6507-493a-867d-880a4c5f1395\") " Oct 30 22:13:32.928034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927790 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.928034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927802 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.928034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927893 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.928034 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.927914 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.928314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928104 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.928314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928127 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.928314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928155 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.928314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928165 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.928314 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928173 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:32.928557 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.928415 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.930398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930376 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:32.930498 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930380 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:32.930579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930561 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz" pod "eaaed7af-6507-493a-867d-880a4c5f1395" (UID: "eaaed7af-6507-493a-867d-880a4c5f1395"). InnerVolumeSpecName "kube-api-access-fhtjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.930644 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930558 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:32.930848 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930824 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "kube-api-access-stp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:32.930848 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:32.930828 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm" pod "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" (UID: "d6ba82b0-c5c1-48bc-be00-0ca072a80ef5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029290 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029328 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029345 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stp4b\" (UniqueName: \"kubernetes.io/projected/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-kube-api-access-stp4b\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029368 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029382 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhtjz\" (UniqueName: \"kubernetes.io/projected/eaaed7af-6507-493a-867d-880a4c5f1395-kube-api-access-fhtjz\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029422 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaaed7af-6507-493a-867d-880a4c5f1395-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.029863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.029676 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaaed7af-6507-493a-867d-880a4c5f1395-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:33.514427 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:13:33.513060 4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaaed7af_6507_493a_867d_880a4c5f1395.slice/crio-90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaaed7af_6507_493a_867d_880a4c5f1395.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ba82b0_c5c1_48bc_be00_0ca072a80ef5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ba82b0_c5c1_48bc_be00_0ca072a80ef5.slice/crio-4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46\": RecentStats: unable to find data in memory cache]" Oct 30 22:13:33.870257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.870226 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" Oct 30 22:13:33.870257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.870241 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9" event={"ID":"eaaed7af-6507-493a-867d-880a4c5f1395","Type":"ContainerDied","Data":"90e07831ffc8f2f83b50e91afb50bd6c777c3619a2bbae47871124fc7fd77a86"} Oct 30 22:13:33.870852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.870296 4525 scope.go:122] "RemoveContainer" containerID="4a0067a264ebfc0732de5ad83d142dae1b3233702f39c44f20e5902fba82e3de" Oct 30 22:13:33.871775 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.871742 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" event={"ID":"d6ba82b0-c5c1-48bc-be00-0ca072a80ef5","Type":"ContainerDied","Data":"4daf590cecc09bf264469331db2a5bdeddbf448f15ba0f82471265620ce91c46"} Oct 30 22:13:33.871775 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.871762 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh" Oct 30 22:13:33.878806 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:33.878789 4525 scope.go:122] "RemoveContainer" containerID="1f53bd9f71ac5d98ba60e7f3c56f5f6dbad7840db85d0bcfd92789c6e13977b7" Oct 30 22:13:34.054107 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:34.054070 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 22:13:34.104056 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:34.104026 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-2-w6-69e4b2-kserve-648785df96-zcgl9"] Oct 30 22:13:34.215929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:34.215842 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 22:13:34.219992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:34.219964 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-69e4b2/distinct-w6-69e4b2-kserve-7dbfc47fd7-9z9jh"] Oct 30 22:13:35.394517 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:13:35.394476 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:13:35.471490 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:35.471458 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" path="/var/lib/kubelet/pods/d6ba82b0-c5c1-48bc-be00-0ca072a80ef5/volumes" Oct 30 22:13:35.471910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:35.471896 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" path="/var/lib/kubelet/pods/eaaed7af-6507-493a-867d-880a4c5f1395/volumes" Oct 30 22:13:44.377244 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:44.377207 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:13:58.274125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:13:58.274087 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:14:05.398869 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:14:05.398828 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:14:07.917209 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:07.917180 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 22:14:07.917895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:07.917405 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-685458bf9d-bz826" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" containerID="cri-o://31c0773591e788ae42c4bb4cd39e101c0c4cb4dc6d5b0c2b09081e298d907061" gracePeriod=30 Oct 30 22:14:08.037017 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.036977 4525 generic.go:356] "Generic (PLEG): container finished" podID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerID="31c0773591e788ae42c4bb4cd39e101c0c4cb4dc6d5b0c2b09081e298d907061" exitCode=0 Oct 30 22:14:08.037189 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.037013 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerDied","Data":"31c0773591e788ae42c4bb4cd39e101c0c4cb4dc6d5b0c2b09081e298d907061"} Oct 30 22:14:08.037189 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.037083 4525 scope.go:122] "RemoveContainer" containerID="169d99f8d9d711710aec7cda8317baed6f984ede7ef7d0a6808e27ac511e1f4f" Oct 30 22:14:08.171332 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.171266 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 22:14:08.270277 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.270233 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\") pod \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " Oct 30 22:14:08.270454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.270291 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\") pod \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " Oct 30 22:14:08.270454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.270338 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\" (UniqueName: \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\") pod \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\" (UID: \"67f863a5-33e2-4314-a6fc-c95d748ab8a9\") " Oct 30 22:14:08.272725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.272683 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p" pod "67f863a5-33e2-4314-a6fc-c95d748ab8a9" (UID: "67f863a5-33e2-4314-a6fc-c95d748ab8a9"). InnerVolumeSpecName "kube-api-access-ggl2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:08.272725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.272709 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert" pod "67f863a5-33e2-4314-a6fc-c95d748ab8a9" (UID: "67f863a5-33e2-4314-a6fc-c95d748ab8a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:08.272892 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.272726 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs" pod "67f863a5-33e2-4314-a6fc-c95d748ab8a9" (UID: "67f863a5-33e2-4314-a6fc-c95d748ab8a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:08.371934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.371890 4525 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-webhook-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:08.371934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.371932 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67f863a5-33e2-4314-a6fc-c95d748ab8a9-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:08.372144 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.371949 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggl2p\" (UniqueName: \"kubernetes.io/projected/67f863a5-33e2-4314-a6fc-c95d748ab8a9-kube-api-access-ggl2p\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:08.531499 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:08.531415 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:14:09.044242 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.044207 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-bz826" Oct 30 22:14:09.044754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.044210 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-bz826" event={"ID":"67f863a5-33e2-4314-a6fc-c95d748ab8a9","Type":"ContainerDied","Data":"79ff034ddfdc1de9aa4c6fda417f8411279574707491e453d2a0254689582ef9"} Oct 30 22:14:09.044754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.044345 4525 scope.go:122] "RemoveContainer" containerID="31c0773591e788ae42c4bb4cd39e101c0c4cb4dc6d5b0c2b09081e298d907061" Oct 30 22:14:09.209871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.209838 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 22:14:09.259869 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.259838 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-685458bf9d-bz826"] Oct 30 22:14:09.472989 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:09.472957 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" path="/var/lib/kubelet/pods/67f863a5-33e2-4314-a6fc-c95d748ab8a9/volumes" Oct 30 22:14:18.923056 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923020 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923394 4525 cpu_manager.go:414] "containerMap: removing container" podUID="30798ac3-f1bc-40bc-bc83-5914da439947" containerName="maas-api" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923406 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30798ac3-f1bc-40bc-bc83-5914da439947" containerName="maas-api" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923419 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923425 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923438 4525 cpu_manager.go:414] "containerMap: removing container" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923445 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923454 4525 cpu_manager.go:414] "containerMap: removing container" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" Oct 30 22:14:18.923460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923459 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923467 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923472 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923478 4525 cpu_manager.go:414] "containerMap: removing container" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923483 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923490 4525 cpu_manager.go:414] "containerMap: removing container" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923494 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923501 4525 cpu_manager.go:414] "containerMap: removing container" podUID="3c2f241c-8757-4636-a978-34ef53274d57" containerName="main" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923507 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2f241c-8757-4636-a978-34ef53274d57" containerName="main" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923519 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923524 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923531 4525 cpu_manager.go:414] "containerMap: removing container" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923535 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923542 4525 cpu_manager.go:414] "containerMap: removing container" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923546 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923553 4525 cpu_manager.go:414] "containerMap: removing container" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923558 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923563 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923567 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923574 4525 cpu_manager.go:414] "containerMap: removing container" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923578 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923590 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923598 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923619 4525 cpu_manager.go:414] "containerMap: removing container" podUID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923624 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923635 4525 cpu_manager.go:414] "containerMap: removing container" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923640 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923647 4525 cpu_manager.go:414] "containerMap: removing container" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923652 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923658 4525 cpu_manager.go:414] "containerMap: removing container" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923663 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923673 4525 cpu_manager.go:414] "containerMap: removing container" podUID="1672e201-f9de-436b-a775-f0617c9dbfbd" containerName="payload-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923680 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1672e201-f9de-436b-a775-f0617c9dbfbd" containerName="payload-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923689 4525 cpu_manager.go:414] "containerMap: removing container" podUID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923693 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923751 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="30798ac3-f1bc-40bc-bc83-5914da439947" containerName="maas-api" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923762 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ee4b7e-8af1-4e33-9811-a71f24cb49d5" containerName="payload-pre-processing" Oct 30 22:14:18.923753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923768 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="987f3950-569e-47e4-96b7-995d0289b09a" containerName="payload-processing" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923775 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ba82b0-c5c1-48bc-be00-0ca072a80ef5" containerName="main" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923781 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea6b3ee-cbbf-44af-826e-647792e2cb7c" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923786 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaed7af-6507-493a-867d-880a4c5f1395" containerName="main" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923791 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f340ba07-fb78-4e39-8be7-c1fd6deac8ee" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923797 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923802 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2f241c-8757-4636-a978-34ef53274d57" containerName="main" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923807 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b544a7d-7df3-4799-841c-f5069200e931" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923813 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e4a174-bcd8-46e7-a1f7-4daa66e70c9e" containerName="maas-api" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923818 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d205b6-099b-47c6-a8b3-c6e18380473f" containerName="payload-pre-processing" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923823 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923830 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923837 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4f334c-f3fd-4052-926f-c10c16bd7049" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923843 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923847 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39db119-794c-469d-a941-8f28110bd3bd" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923852 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0721afed-862b-4402-8f71-0dea67fd2bbb" containerName="istio-proxy" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923858 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe516fb-955b-4a93-8dd8-63c2bd851b8c" containerName="maas-api" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923864 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="508779a4-73f5-48e8-869d-4e54f50fef0a" containerName="payload-pre-processing" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923869 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1672e201-f9de-436b-a775-f0617c9dbfbd" containerName="payload-processing" Oct 30 22:14:18.924853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.923876 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e1b468-709d-4e9e-b975-2d732b6bdc40" containerName="maas-api" Oct 30 22:14:18.928223 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.928206 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:18.971677 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.971635 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:18.971677 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.971684 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9b8n\" (UniqueName: \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:18.971907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.971835 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:18.981176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:18.981151 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:14:19.031013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.030980 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:14:19.031186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.031034 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:14:19.031186 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.031126 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cl4bb\"" Oct 30 22:14:19.032856 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.032835 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:19.072251 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.072220 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.072443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.072292 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.072443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.072331 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9b8n\" (UniqueName: \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.074861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.074832 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.075301 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.075276 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.134941 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.134902 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9b8n\" (UniqueName: \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\") pod \"maas-controller-685458bf9d-49v65\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.240575 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.240469 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:19.616434 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.616410 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:19.618788 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:14:19.618748 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14645d3d_c7ed_43fb_bf9c_de8c8a2aba06.slice/crio-d680b5ca48849eb15b203f92f35ea7e30c33c7ce7f793e4618d2411c48e1ed6f WatchSource:0}: Error finding container d680b5ca48849eb15b203f92f35ea7e30c33c7ce7f793e4618d2411c48e1ed6f: Status 404 returned error can't find the container with id d680b5ca48849eb15b203f92f35ea7e30c33c7ce7f793e4618d2411c48e1ed6f Oct 30 22:14:19.620154 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:19.620136 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:14:20.095721 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:20.095689 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-49v65" event={"ID":"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06","Type":"ContainerStarted","Data":"d680b5ca48849eb15b203f92f35ea7e30c33c7ce7f793e4618d2411c48e1ed6f"} Oct 30 22:14:21.101426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:21.101388 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-49v65" event={"ID":"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06","Type":"ContainerStarted","Data":"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd"} Oct 30 22:14:21.101915 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:21.101499 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:21.261490 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:21.261436 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-685458bf9d-49v65" podStartSLOduration=2.448855602 podStartE2EDuration="3.26141932s" totalImagesPullingTime="812.563718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:21.259251531 +0000 UTC m=+3085.933298070" watchObservedRunningTime="2026-10-30 22:14:21.26141932 +0000 UTC m=+3085.935465855" Oct 30 22:14:30.267580 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.267534 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:14:30.268125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268094 4525 cpu_manager.go:414] "containerMap: removing container" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:30.268125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268113 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f863a5-33e2-4314-a6fc-c95d748ab8a9" containerName="manager" Oct 30 22:14:30.268261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268132 4525 cpu_manager.go:414] "containerMap: removing container" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:30.268261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268140 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:30.268261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268241 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5327b3bf-027c-43b3-a35e-3ed832f71182" containerName="istio-proxy" Oct 30 22:14:30.268261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.268253 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c1417d-e609-42c9-a9f7-afde6ed45e5c" containerName="istio-proxy" Oct 30 22:14:30.271397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.271374 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.384757 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.384713 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.384757 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.384760 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.384849 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.384904 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.384972 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.385006 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385193 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.385035 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lndvw\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385193 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.385089 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.385193 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.385115 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.482416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.482379 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:14:30.486146 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486114 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486173 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486200 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486234 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486266 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486321 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486303 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486588 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486345 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lndvw\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486588 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486374 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.486588 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.486419 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.487289 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.487264 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.487443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.487419 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.487560 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.487375 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.487715 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.487594 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.488081 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.488055 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.490264 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.490242 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.490760 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.490734 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.549338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.549260 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.549953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.549927 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lndvw\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\") pod \"maas-default-gateway-openshift-default-75d7c785f-mndl2\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:30.584722 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:30.584681 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:31.012051 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.012025 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:14:31.013947 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:14:31.013919 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d47965_7123_48b0_9658_28993d573c42.slice/crio-4ac3b32e0b47171f6cee5141f63f9ccd496adac6b4b8a134032860d112b32eb3 WatchSource:0}: Error finding container 4ac3b32e0b47171f6cee5141f63f9ccd496adac6b4b8a134032860d112b32eb3: Status 404 returned error can't find the container with id 4ac3b32e0b47171f6cee5141f63f9ccd496adac6b4b8a134032860d112b32eb3 Oct 30 22:14:31.016012 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.015980 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:31.016084 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.016049 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:31.016084 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.016077 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:31.148010 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.147974 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" event={"ID":"46d47965-7123-48b0-9658-28993d573c42","Type":"ContainerStarted","Data":"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9"} Oct 30 22:14:31.148191 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.148028 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" event={"ID":"46d47965-7123-48b0-9658-28993d573c42","Type":"ContainerStarted","Data":"4ac3b32e0b47171f6cee5141f63f9ccd496adac6b4b8a134032860d112b32eb3"} Oct 30 22:14:31.269926 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.269811 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podStartSLOduration=1.269793642 podStartE2EDuration="1.269793642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:31.267787632 +0000 UTC m=+3095.941834169" watchObservedRunningTime="2026-10-30 22:14:31.269793642 +0000 UTC m=+3095.943840253" Oct 30 22:14:31.584861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:31.584779 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:32.108910 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:32.108877 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:32.587284 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:32.587242 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:32.587724 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:32.587317 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:33.585238 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:33.585204 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:33.585396 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:33.585272 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:34.585204 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:34.585165 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:34.585637 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:34.585237 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:35.395342 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:14:35.395299 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:14:35.585259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:35.585223 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:35.585725 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:35.585288 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:36.585837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:36.585753 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:36.585837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:36.585819 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:37.585087 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:37.585050 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:37.585307 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:37.585113 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:38.585264 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:38.585217 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 22:14:38.585689 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:38.585291 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 22:14:39.589763 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:39.589729 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:39.590323 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:39.590033 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:39.590778 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:39.590760 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:14:44.371218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:44.371175 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:14:45.377360 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.377328 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:14:45.381836 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.381814 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.487972 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.487935 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:14:45.550853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.550804 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.550864 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551049 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551028 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551089 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551135 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551229 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551278 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551353 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.551518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.551397 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtkb\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652583 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652651 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652688 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652718 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652748 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652774 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtkb\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652854 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652887 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.652992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.652914 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.653344 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.653154 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.653344 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.653174 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.653453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.653407 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.653564 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.653526 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.653800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.653719 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.655333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.655310 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.655736 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.655711 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.714428 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.714387 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.714649 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.714431 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtkb\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\") pod \"maas-default-gateway-openshift-default-75d7c785f-5xcrz\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:45.996935 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:45.996848 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:46.425110 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:46.425060 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:14:46.427447 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:14:46.427418 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e06632_8ee0_4694_9de1_544e13d57691.slice/crio-0914691adb90cea4e221dd7bc812143b8c90375ca143cf446ec5a66c50c6ddea WatchSource:0}: Error finding container 0914691adb90cea4e221dd7bc812143b8c90375ca143cf446ec5a66c50c6ddea: Status 404 returned error can't find the container with id 0914691adb90cea4e221dd7bc812143b8c90375ca143cf446ec5a66c50c6ddea Oct 30 22:14:46.429758 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:46.429724 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:46.429843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:46.429791 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:46.429843 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:46.429820 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:47.251093 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.251049 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" event={"ID":"e5e06632-8ee0-4694-9de1-544e13d57691","Type":"ContainerStarted","Data":"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932"} Oct 30 22:14:47.252670 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.252639 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" event={"ID":"e5e06632-8ee0-4694-9de1-544e13d57691","Type":"ContainerStarted","Data":"0914691adb90cea4e221dd7bc812143b8c90375ca143cf446ec5a66c50c6ddea"} Oct 30 22:14:47.373835 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.373787 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podStartSLOduration=2.373772439 podStartE2EDuration="2.373772439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:47.373097019 +0000 UTC m=+3112.047143556" watchObservedRunningTime="2026-10-30 22:14:47.373772439 +0000 UTC m=+3112.047818974" Oct 30 22:14:47.997066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.997035 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:47.998767 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.998740 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:47.998906 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:47.998793 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:48.998197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:48.998157 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:48.998666 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:48.998217 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:49.998088 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:49.998046 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:49.998269 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:49.998106 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:50.997439 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:50.997397 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:50.997762 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:50.997453 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:51.499805 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.499762 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:51.500241 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.500022 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-685458bf9d-49v65" podUID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" containerName="manager" containerID="cri-o://59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd" gracePeriod=30 Oct 30 22:14:51.780801 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.780773 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:51.808979 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.808950 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\" (UniqueName: \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\") pod \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " Oct 30 22:14:51.809131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.809021 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\") pod \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " Oct 30 22:14:51.809131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.809124 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\") pod \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\" (UID: \"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06\") " Oct 30 22:14:51.811234 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.811202 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n" pod "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" (UID: "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06"). InnerVolumeSpecName "kube-api-access-h9b8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:51.811419 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.811226 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert" pod "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" (UID: "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:51.811525 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.811470 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs" pod "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" (UID: "14645d3d-c7ed-43fb-bf9c-de8c8a2aba06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:51.910368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.910315 4525 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-webhook-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.910368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.910360 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.910368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.910373 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9b8n\" (UniqueName: \"kubernetes.io/projected/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06-kube-api-access-h9b8n\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:51.997980 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.997945 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:51.998147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:51.998002 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:52.277948 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.277916 4525 generic.go:356] "Generic (PLEG): container finished" podID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" containerID="59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd" exitCode=0 Oct 30 22:14:52.278148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.277993 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-49v65" event={"ID":"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06","Type":"ContainerDied","Data":"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd"} Oct 30 22:14:52.278148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.278043 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-49v65" Oct 30 22:14:52.278148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.278081 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-49v65" event={"ID":"14645d3d-c7ed-43fb-bf9c-de8c8a2aba06","Type":"ContainerDied","Data":"d680b5ca48849eb15b203f92f35ea7e30c33c7ce7f793e4618d2411c48e1ed6f"} Oct 30 22:14:52.278148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.278108 4525 scope.go:122] "RemoveContainer" containerID="59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd" Oct 30 22:14:52.287084 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.287064 4525 scope.go:122] "RemoveContainer" containerID="59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd" Oct 30 22:14:52.287364 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:14:52.287345 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd\": container with ID starting with 59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd not found: ID does not exist" containerID="59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd" Oct 30 22:14:52.287417 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.287375 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd"} err="failed to get container status \"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd\": rpc error: code = NotFound desc = could not find container \"59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd\": container with ID starting with 59bbcc95ceb84beb0f9e568a27e3a7a70926c7dab700d07e96cee5e7ecf5bdbd not found: ID does not exist" Oct 30 22:14:52.452929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.452894 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:52.457863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.457840 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-685458bf9d-49v65"] Oct 30 22:14:52.998188 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.998145 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 22:14:52.998587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:52.998229 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 22:14:53.473339 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.473303 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" path="/var/lib/kubelet/pods/14645d3d-c7ed-43fb-bf9c-de8c8a2aba06/volumes" Oct 30 22:14:53.883319 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.883289 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685458bf9d-4hg6m"] Oct 30 22:14:53.883720 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.883706 4525 cpu_manager.go:414] "containerMap: removing container" podUID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" containerName="manager" Oct 30 22:14:53.883772 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.883722 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" containerName="manager" Oct 30 22:14:53.883814 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.883792 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="14645d3d-c7ed-43fb-bf9c-de8c8a2aba06" containerName="manager" Oct 30 22:14:53.888312 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.888291 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:53.930668 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.930636 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-webhook-cert\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:53.930852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.930684 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6vnv\" (UniqueName: \"kubernetes.io/projected/72e7e8d8-b262-47ee-8e76-85f72c893624-kube-api-access-x6vnv\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:53.930852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.930781 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-metrics-certs\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:53.991505 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.991471 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:14:53.991768 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.991593 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:14:53.991768 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.991638 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cl4bb\"" Oct 30 22:14:53.991768 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.991747 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:14:53.993508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:53.993486 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-4hg6m"] Oct 30 22:14:54.031896 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.031860 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-webhook-cert\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.032401 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.031924 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6vnv\" (UniqueName: \"kubernetes.io/projected/72e7e8d8-b262-47ee-8e76-85f72c893624-kube-api-access-x6vnv\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.032401 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.031951 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-metrics-certs\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.034791 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.034762 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-webhook-cert\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.034916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.034762 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72e7e8d8-b262-47ee-8e76-85f72c893624-metrics-certs\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.094533 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.094495 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6vnv\" (UniqueName: \"kubernetes.io/projected/72e7e8d8-b262-47ee-8e76-85f72c893624-kube-api-access-x6vnv\") pod \"maas-controller-685458bf9d-4hg6m\" (UID: \"72e7e8d8-b262-47ee-8e76-85f72c893624\") " pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.203216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.203116 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:54.585071 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.584979 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685458bf9d-4hg6m"] Oct 30 22:14:54.616978 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:14:54.616947 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e7e8d8_b262_47ee_8e76_85f72c893624.slice/crio-c41a7878e45eaf985c3c14bddfa35994061f727db9b2808ce0cd40450030e6cd WatchSource:0}: Error finding container c41a7878e45eaf985c3c14bddfa35994061f727db9b2808ce0cd40450030e6cd: Status 404 returned error can't find the container with id c41a7878e45eaf985c3c14bddfa35994061f727db9b2808ce0cd40450030e6cd Oct 30 22:14:54.683164 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.683117 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:54.683575 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.683549 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:54.684446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:54.684425 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:14:55.294680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:55.294634 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-4hg6m" event={"ID":"72e7e8d8-b262-47ee-8e76-85f72c893624","Type":"ContainerStarted","Data":"c41a7878e45eaf985c3c14bddfa35994061f727db9b2808ce0cd40450030e6cd"} Oct 30 22:14:56.300902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:56.300855 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685458bf9d-4hg6m" event={"ID":"72e7e8d8-b262-47ee-8e76-85f72c893624","Type":"ContainerStarted","Data":"d233f6a108fe2ca90c847c405dedb50c0ed61e8529c6f7a7f59483cfe9683b5a"} Oct 30 22:14:56.301305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:56.301012 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:14:56.421538 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:14:56.421480 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-685458bf9d-4hg6m" podStartSLOduration=2.663782479 podStartE2EDuration="3.421460168s" totalImagesPullingTime="757.677689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:56.419812755 +0000 UTC m=+3121.093859293" watchObservedRunningTime="2026-10-30 22:14:56.421460168 +0000 UTC m=+3121.095506704" Oct 30 22:15:05.395997 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:15:05.395959 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:15:07.307534 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:15:07.307503 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685458bf9d-4hg6m" Oct 30 22:15:33.777733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:15:33.777688 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" podUID="ad6010805d267929c8af4d6a80598dea" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:35.395627 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:15:35.395569 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:15:39.463131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:15:39.463090 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:16:00.948643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:16:00.948590 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:16:05.396823 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:16:05.396777 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:16:12.050949 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:16:12.050867 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:16:35.396872 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:16:35.396831 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:16:35.846098 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:16:35.846014 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:16:58.766385 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:16:58.766354 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:17:05.396360 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:17:05.396315 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:17:31.540636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:17:31.539995 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:17:35.396509 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:17:35.396474 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:17:52.175394 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:17:52.175358 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:18:03.445974 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:03.445930 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:18:05.397052 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:18:05.397018 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:18:27.052815 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:27.052779 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:18:34.663172 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:34.662734 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:34.663172 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:34.662813 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:35.395319 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:18:35.395281 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:18:36.931914 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:36.931885 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:18:55.264246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:18:55.264209 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:19:04.663856 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:04.663805 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:04.665083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:04.664568 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:05.395644 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:19:05.395591 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:19:08.051237 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:08.051160 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:19:33.777176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:33.777126 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" podUID="ad6010805d267929c8af4d6a80598dea" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:19:35.395486 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:19:35.395443 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:19:37.837058 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:37.837023 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:19:40.642634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:40.641708 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:40.642634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:40.641802 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:40.728745 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:40.728714 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:19:40.728927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:40.728783 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:19:59.957154 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:19:59.957113 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:20:05.396979 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:20:05.396935 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:20:13.149636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:20:13.149585 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:20:23.948883 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:20:23.948850 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:20:35.395783 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:20:35.395744 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:20:54.560059 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:20:54.560023 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:21:04.855534 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:21:04.855500 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:21:05.396384 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:21:05.396343 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:21:18.645753 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:21:18.645715 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:21:29.314934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:21:29.314899 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:21:35.395762 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:21:35.395713 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:22:00.549831 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:00.549794 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:22:05.396025 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:22:05.395984 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:22:11.236857 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:11.236775 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:22:28.753418 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:28.753382 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:22:35.395508 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:22:35.395470 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:22:40.036346 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:40.036306 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:22:56.909852 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:56.909812 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:22:56.914118 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:56.914092 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:56.968849 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:56.968808 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1e5514\"" Oct 30 22:22:56.974360 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:56.974327 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:56.974519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:56.974464 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kr9\" (UniqueName: \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.018770 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.018736 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-51b49da0-dockercfg-gh9xp\"" Oct 30 22:22:57.020654 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.020626 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:22:57.075342 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.075312 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kr9\" (UniqueName: \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.075548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.075378 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.076078 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.076057 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.092461 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.092434 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:22:57.095981 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.095967 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.136717 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.136689 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kr9\" (UniqueName: \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\") pod \"payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.156548 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.156520 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:22:57.176125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.176049 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.176259 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.176149 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldn9\" (UniqueName: \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.232421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.232391 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.276845 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.276801 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.277019 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.276895 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldn9\" (UniqueName: \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.277476 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.277457 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.338320 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.338295 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldn9\" (UniqueName: \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\") pod \"payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.409537 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.409500 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:57.604527 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.604490 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:22:57.606708 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:22:57.606671 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a96aeef_b9b8_43a2_8aca_bfc6d6e5e2d5.slice/crio-1c22a16ef9ab30e024d8929ab6ec3921a4873f5ce623c61042fe5387d945088e WatchSource:0}: Error finding container 1c22a16ef9ab30e024d8929ab6ec3921a4873f5ce623c61042fe5387d945088e: Status 404 returned error can't find the container with id 1c22a16ef9ab30e024d8929ab6ec3921a4873f5ce623c61042fe5387d945088e Oct 30 22:22:57.715407 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.715326 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" event={"ID":"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5","Type":"ContainerStarted","Data":"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746"} Oct 30 22:22:57.715407 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.715372 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" event={"ID":"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5","Type":"ContainerStarted","Data":"1c22a16ef9ab30e024d8929ab6ec3921a4873f5ce623c61042fe5387d945088e"} Oct 30 22:22:57.715407 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.715402 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:22:57.785089 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.785061 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:22:57.786939 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:22:57.786913 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb6fcc8_5e2a_401e_b123_2d0599e5e68e.slice/crio-6bf99551da18f9123b8377e77bef86ad7994d8e4eab8d9832bfaaf92a3168ca3 WatchSource:0}: Error finding container 6bf99551da18f9123b8377e77bef86ad7994d8e4eab8d9832bfaaf92a3168ca3: Status 404 returned error can't find the container with id 6bf99551da18f9123b8377e77bef86ad7994d8e4eab8d9832bfaaf92a3168ca3 Oct 30 22:22:57.833280 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:57.833220 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" podStartSLOduration=1.833201199 podStartE2EDuration="1.833201199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:57.83092099 +0000 UTC m=+3602.504967524" watchObservedRunningTime="2026-10-30 22:22:57.833201199 +0000 UTC m=+3602.507247736" Oct 30 22:22:58.721639 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:58.721585 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" event={"ID":"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e","Type":"ContainerStarted","Data":"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64"} Oct 30 22:22:58.722055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:58.721663 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" event={"ID":"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e","Type":"ContainerStarted","Data":"6bf99551da18f9123b8377e77bef86ad7994d8e4eab8d9832bfaaf92a3168ca3"} Oct 30 22:22:58.722055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:58.721788 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:22:58.838155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:58.838104 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" podStartSLOduration=1.8380898989999999 podStartE2EDuration="1.838089899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:58.836591426 +0000 UTC m=+3603.510637962" watchObservedRunningTime="2026-10-30 22:22:58.838089899 +0000 UTC m=+3603.512136428" Oct 30 22:22:59.203248 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.203212 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:22:59.206663 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.206631 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.259334 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.259300 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1e5514-serving-cert\"" Oct 30 22:22:59.297102 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.297061 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.297291 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.297180 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.297291 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.297221 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tg6\" (UniqueName: \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.297379 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.297350 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.310833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.310796 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:22:59.398705 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.398662 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.398895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.398743 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.398895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.398778 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tg6\" (UniqueName: \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.398895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.398813 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.398895 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:22:59.398821 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-serving-cert: secret "maas-api-e2e-worker-single-main-1e5514-serving-cert" not found Oct 30 22:22:59.399045 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:22:59.398897 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls podName:fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa nodeName:}" failed. No retries permitted until 2026-10-30 22:22:59.898876123 +0000 UTC m=+3604.572922645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls") pod "maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" (UID: "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa") : secret "maas-api-e2e-worker-single-main-1e5514-serving-cert" not found Oct 30 22:22:59.399478 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.399461 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.401477 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.401459 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.470929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.470843 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tg6\" (UniqueName: \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.909041 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.908995 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:22:59.911559 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:22:59.911536 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:23:00.119446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:00.119403 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:23:00.494273 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:00.494248 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:23:00.497096 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:23:00.497060 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3c175f_e1b5_49ec_b7e7_a0ce4b8bdffa.slice/crio-74b77cdd55899d58b83c28af3f428f9f0cd2d43bd77c697f81b7b18baf531025 WatchSource:0}: Error finding container 74b77cdd55899d58b83c28af3f428f9f0cd2d43bd77c697f81b7b18baf531025: Status 404 returned error can't find the container with id 74b77cdd55899d58b83c28af3f428f9f0cd2d43bd77c697f81b7b18baf531025 Oct 30 22:23:00.498556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:00.498539 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:23:00.734213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:00.734183 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" event={"ID":"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa","Type":"ContainerStarted","Data":"74b77cdd55899d58b83c28af3f428f9f0cd2d43bd77c697f81b7b18baf531025"} Oct 30 22:23:01.740862 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:01.740821 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" event={"ID":"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa","Type":"ContainerStarted","Data":"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d"} Oct 30 22:23:01.741290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:01.740892 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:23:01.861718 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:01.861658 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" podStartSLOduration=2.061244234 podStartE2EDuration="2.861642289s" totalImagesPullingTime="800.398055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:01.859982476 +0000 UTC m=+3606.534029027" watchObservedRunningTime="2026-10-30 22:23:01.861642289 +0000 UTC m=+3606.535688824" Oct 30 22:23:05.395879 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:23:05.395844 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:23:07.524985 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:07.524947 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:23:07.752158 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:07.752123 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:23:08.725799 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:08.725769 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:23:09.731643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:09.731589 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:23:22.706992 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.706950 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:23:22.712043 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.712015 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.767060 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.767030 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1e5514\"/\"kube-root-ca.crt\"" Oct 30 22:23:22.767060 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.767030 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1e5514\"/\"openshift-service-ca.crt\"" Oct 30 22:23:22.816822 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.816787 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1e5514\"/\"simulator-single-main-1e5514-kserve-self-signed-certs\"" Oct 30 22:23:22.817127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.817110 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1e5514\"/\"default-dockercfg-kz8kc\"" Oct 30 22:23:22.819313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.819285 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:23:22.872066 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.857487 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.872389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.872354 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ptz\" (UniqueName: \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.872580 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.872561 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.872745 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.872728 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.872950 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.872922 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.873096 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.872984 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.973946 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.973995 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974041 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974059 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ptz\" (UniqueName: \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974090 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974107 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974440 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974519 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974451 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.974655 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.974552 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.976748 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.976724 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:22.977023 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:22.977001 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:23.036234 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.036192 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ptz\" (UniqueName: \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\") pod \"simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:23.045039 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.044999 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:23.420734 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.420710 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:23:23.423074 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:23:23.423043 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99370a30_2434_4128_9d88_bcd67a5a213c.slice/crio-b5a509aea4fdbd47fef53f4dbf4b448b04a4dcb14eda9ee9ed30c1be34c4c517 WatchSource:0}: Error finding container b5a509aea4fdbd47fef53f4dbf4b448b04a4dcb14eda9ee9ed30c1be34c4c517: Status 404 returned error can't find the container with id b5a509aea4fdbd47fef53f4dbf4b448b04a4dcb14eda9ee9ed30c1be34c4c517 Oct 30 22:23:23.843307 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.843213 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" event={"ID":"99370a30-2434-4128-9d88-bcd67a5a213c","Type":"ContainerStarted","Data":"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e"} Oct 30 22:23:23.843307 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.843263 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" event={"ID":"99370a30-2434-4128-9d88-bcd67a5a213c","Type":"ContainerStarted","Data":"b5a509aea4fdbd47fef53f4dbf4b448b04a4dcb14eda9ee9ed30c1be34c4c517"} Oct 30 22:23:23.988144 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:23.988092 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" podStartSLOduration=1.9880781 podStartE2EDuration="1.9880781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:23.986368917 +0000 UTC m=+3628.660415453" watchObservedRunningTime="2026-10-30 22:23:23.9880781 +0000 UTC m=+3628.662124635" Oct 30 22:23:24.045571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:24.045534 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:27.620348 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:27.620316 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:23:33.045800 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:33.045759 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:33.058183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:33.058153 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:33.903013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:23:33.902984 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:23:35.395865 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:23:35.395827 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:24:04.653871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:04.653828 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:24:05.396328 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:24:05.396285 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:24:33.646210 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:33.646169 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:24:35.397677 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:24:35.397641 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:24:45.549424 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:45.549390 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:24:52.656462 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:52.656421 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:24:52.761154 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:52.761121 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:24:57.540332 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:57.540299 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:24:57.540785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:57.540533 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerName="maas-api" containerID="cri-o://05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d" gracePeriod=30 Oct 30 22:24:57.657395 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:57.657330 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" containerID="cri-o://c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746" gracePeriod=25 Oct 30 22:24:57.746919 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:57.746881 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.172:8443/health\": dial tcp 10.133.0.172:8443: connect: connection refused" Oct 30 22:24:57.765090 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:57.765008 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" containerID="cri-o://90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64" gracePeriod=25 Oct 30 22:24:58.725348 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.725312 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:24:58.804196 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.804166 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:24:58.938925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.938850 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") pod \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " Oct 30 22:24:58.938925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.938893 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\") pod \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " Oct 30 22:24:58.939171 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.938980 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\") pod \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " Oct 30 22:24:58.939171 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.939055 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\" (UniqueName: \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\") pod \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\" (UID: \"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa\") " Oct 30 22:24:58.939384 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.939343 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca" pod "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" (UID: "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:58.941467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.941443 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs" pod "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" (UID: "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:58.941626 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.941499 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls" pod "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" (UID: "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:58.941626 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:58.941526 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6" pod "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" (UID: "fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa"). InnerVolumeSpecName "kube-api-access-c7tg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:59.040315 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.040284 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.040315 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.040314 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tg6\" (UniqueName: \"kubernetes.io/projected/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-kube-api-access-c7tg6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.040315 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.040324 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.040590 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.040333 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:59.299922 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.299829 4525 generic.go:356] "Generic (PLEG): container finished" podID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerID="05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d" exitCode=0 Oct 30 22:24:59.299922 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.299880 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" Oct 30 22:24:59.300150 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.299915 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" event={"ID":"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa","Type":"ContainerDied","Data":"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d"} Oct 30 22:24:59.300150 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.299974 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp" event={"ID":"fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa","Type":"ContainerDied","Data":"74b77cdd55899d58b83c28af3f428f9f0cd2d43bd77c697f81b7b18baf531025"} Oct 30 22:24:59.300150 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.300002 4525 scope.go:122] "RemoveContainer" containerID="05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d" Oct 30 22:24:59.309442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.309422 4525 scope.go:122] "RemoveContainer" containerID="05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d" Oct 30 22:24:59.309750 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:24:59.309727 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d\": container with ID starting with 05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d not found: ID does not exist" containerID="05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d" Oct 30 22:24:59.309801 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.309759 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d"} err="failed to get container status \"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d\": rpc error: code = NotFound desc = could not find container \"05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d\": container with ID starting with 05d621d831245752a1c95456cedd16083728f44bfba417b2097d3bce9d7a6e9d not found: ID does not exist" Oct 30 22:24:59.465491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.465455 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:24:59.471460 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.471432 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1e5514-7d84b6db95-svzpp"] Oct 30 22:24:59.730190 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:24:59.730156 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:25:00.570247 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.570201 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" podUID="99370a30-2434-4128-9d88-bcd67a5a213c" containerName="main" containerID="cri-o://f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e" gracePeriod=45 Oct 30 22:25:00.825325 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.825266 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:25:00.964842 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.964804 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\" (UniqueName: \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965065 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.964877 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965065 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.964934 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965065 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.965053 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.965101 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.965134 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\" (UniqueName: \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\") pod \"99370a30-2434-4128-9d88-bcd67a5a213c\" (UID: \"99370a30-2434-4128-9d88-bcd67a5a213c\") " Oct 30 22:25:00.965953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.965738 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.965953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.965775 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.966241 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.966206 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:00.970132 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.970102 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:00.970337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.970317 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "kube-api-access-v9ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:00.970982 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:00.970958 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm" pod "99370a30-2434-4128-9d88-bcd67a5a213c" (UID: "99370a30-2434-4128-9d88-bcd67a5a213c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.067397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067330 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.067397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067372 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.067397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067386 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99370a30-2434-4128-9d88-bcd67a5a213c-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.067397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067401 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ptz\" (UniqueName: \"kubernetes.io/projected/99370a30-2434-4128-9d88-bcd67a5a213c-kube-api-access-v9ptz\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.067948 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067415 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.067948 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.067436 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99370a30-2434-4128-9d88-bcd67a5a213c-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.311522 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.311484 4525 generic.go:356] "Generic (PLEG): container finished" podID="99370a30-2434-4128-9d88-bcd67a5a213c" containerID="f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e" exitCode=0 Oct 30 22:25:01.311716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.311532 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" Oct 30 22:25:01.311716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.311562 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" event={"ID":"99370a30-2434-4128-9d88-bcd67a5a213c","Type":"ContainerDied","Data":"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e"} Oct 30 22:25:01.311716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.311627 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm" event={"ID":"99370a30-2434-4128-9d88-bcd67a5a213c","Type":"ContainerDied","Data":"b5a509aea4fdbd47fef53f4dbf4b448b04a4dcb14eda9ee9ed30c1be34c4c517"} Oct 30 22:25:01.311716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.311647 4525 scope.go:122] "RemoveContainer" containerID="f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e" Oct 30 22:25:01.320791 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.320775 4525 scope.go:122] "RemoveContainer" containerID="f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e" Oct 30 22:25:01.321088 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:25:01.321069 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e\": container with ID starting with f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e not found: ID does not exist" containerID="f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e" Oct 30 22:25:01.321131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.321099 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e"} err="failed to get container status \"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e\": rpc error: code = NotFound desc = could not find container \"f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e\": container with ID starting with f1c83f2324720a0fad67aaac2bbb6037f6b505e33047fb219cff9bfc32a6a87e not found: ID does not exist" Oct 30 22:25:01.471846 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.471812 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" path="/var/lib/kubelet/pods/fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa/volumes" Oct 30 22:25:01.475920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.475893 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:25:01.480446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:01.480420 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1e5514/simulator-single-main-1e5514-kserve-6b47d69f9-ff4mm"] Oct 30 22:25:02.346951 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:02.346918 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:25:03.471994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:03.471965 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99370a30-2434-4128-9d88-bcd67a5a213c" path="/var/lib/kubelet/pods/99370a30-2434-4128-9d88-bcd67a5a213c/volumes" Oct 30 22:25:05.397136 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:25:05.397099 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:25:08.724302 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:08.724216 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:25:09.730079 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:09.730039 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:25:17.906807 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:17.906780 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:25:18.012528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.012502 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:25:18.037083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.037057 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\") pod \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " Oct 30 22:25:18.037270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.037241 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\" (UniqueName: \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\") pod \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\" (UID: \"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5\") " Oct 30 22:25:18.037415 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.037386 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume" pod "2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" (UID: "2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:18.037707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.037686 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:18.039405 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.039386 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9" pod "2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" (UID: "2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5"). InnerVolumeSpecName "kube-api-access-x4kr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:18.138395 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.138362 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\" (UniqueName: \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\") pod \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " Oct 30 22:25:18.138395 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.138404 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\") pod \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\" (UID: \"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e\") " Oct 30 22:25:18.138664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.138642 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kr9\" (UniqueName: \"kubernetes.io/projected/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5-kube-api-access-x4kr9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:18.138822 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.138793 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume" pod "9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" (UID: "9bb6fcc8-5e2a-401e-b123-2d0599e5e68e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:18.141234 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.141202 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9" pod "9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" (UID: "9bb6fcc8-5e2a-401e-b123-2d0599e5e68e"). InnerVolumeSpecName "kube-api-access-kldn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:18.239270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.239186 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kldn9\" (UniqueName: \"kubernetes.io/projected/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-kube-api-access-kldn9\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:18.239270 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.239216 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:18.399254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.399219 4525 generic.go:356] "Generic (PLEG): container finished" podID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerID="c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746" exitCode=0 Oct 30 22:25:18.399441 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.399265 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" Oct 30 22:25:18.399441 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.399310 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" event={"ID":"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5","Type":"ContainerDied","Data":"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746"} Oct 30 22:25:18.399441 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.399359 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5" event={"ID":"2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5","Type":"ContainerDied","Data":"1c22a16ef9ab30e024d8929ab6ec3921a4873f5ce623c61042fe5387d945088e"} Oct 30 22:25:18.399441 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.399384 4525 scope.go:122] "RemoveContainer" containerID="c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746" Oct 30 22:25:18.400726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.400706 4525 generic.go:356] "Generic (PLEG): container finished" podID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerID="90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64" exitCode=0 Oct 30 22:25:18.401022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.400736 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" event={"ID":"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e","Type":"ContainerDied","Data":"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64"} Oct 30 22:25:18.401022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.400754 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" Oct 30 22:25:18.401022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.400787 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg" event={"ID":"9bb6fcc8-5e2a-401e-b123-2d0599e5e68e","Type":"ContainerDied","Data":"6bf99551da18f9123b8377e77bef86ad7994d8e4eab8d9832bfaaf92a3168ca3"} Oct 30 22:25:18.408377 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.408358 4525 scope.go:122] "RemoveContainer" containerID="c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746" Oct 30 22:25:18.408698 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:25:18.408678 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746\": container with ID starting with c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746 not found: ID does not exist" containerID="c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746" Oct 30 22:25:18.408766 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.408708 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746"} err="failed to get container status \"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746\": rpc error: code = NotFound desc = could not find container \"c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746\": container with ID starting with c3dd732ad32d4ba238b32174f6efca688bb91d354e24e0905cc45b0566ed0746 not found: ID does not exist" Oct 30 22:25:18.408766 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.408732 4525 scope.go:122] "RemoveContainer" containerID="90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64" Oct 30 22:25:18.416832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.416817 4525 scope.go:122] "RemoveContainer" containerID="90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64" Oct 30 22:25:18.417125 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:25:18.417106 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64\": container with ID starting with 90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64 not found: ID does not exist" containerID="90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64" Oct 30 22:25:18.417208 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.417132 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64"} err="failed to get container status \"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64\": rpc error: code = NotFound desc = could not find container \"90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64\": container with ID starting with 90d65d7e7c37a59890a0b04f11356a906a2268dfb5777bef2ca164dcc5d38e64 not found: ID does not exist" Oct 30 22:25:18.565148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.565064 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:25:18.569340 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.569314 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1e5514-74df768bwg"] Oct 30 22:25:18.680543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.680512 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:25:18.685101 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:18.685070 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1e5514-7555cc8cdmh5x5"] Oct 30 22:25:19.471770 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:19.471733 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" path="/var/lib/kubelet/pods/2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5/volumes" Oct 30 22:25:19.472257 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:19.472240 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" path="/var/lib/kubelet/pods/9bb6fcc8-5e2a-401e-b123-2d0599e5e68e/volumes" Oct 30 22:25:25.440803 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:25.440772 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:25:35.395187 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:25:35.395149 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:25:36.147311 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:25:36.147278 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:26:05.396124 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:26:05.396081 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:26:35.396199 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:26:35.396159 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:27:05.396187 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:27:05.396140 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:27:07.730255 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:07.730218 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:27:10.640790 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:10.640748 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mndl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:10.641320 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:10.640826 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:10.725758 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:10.725719 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-5xcrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:10.725926 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:10.725788 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:34.542473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:34.542433 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:27:35.396311 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:27:35.396275 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:27:42.944658 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:27:42.944618 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:28:03.784218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:03.784185 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:28:05.395942 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:28:05.395901 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:28:11.899907 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.899808 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900167 4525 cpu_manager.go:414] "containerMap: removing container" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900178 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900198 4525 cpu_manager.go:414] "containerMap: removing container" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerName="maas-api" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900204 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerName="maas-api" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900211 4525 cpu_manager.go:414] "containerMap: removing container" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900216 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900232 4525 cpu_manager.go:414] "containerMap: removing container" podUID="99370a30-2434-4128-9d88-bcd67a5a213c" containerName="main" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900237 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99370a30-2434-4128-9d88-bcd67a5a213c" containerName="main" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900293 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="99370a30-2434-4128-9d88-bcd67a5a213c" containerName="main" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900304 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb6fcc8-5e2a-401e-b123-2d0599e5e68e" containerName="payload-pre-processing" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900311 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3c175f-e1b5-49ec-b7e7-a0ce4b8bdffa" containerName="maas-api" Oct 30 22:28:11.900347 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.900317 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a96aeef-b9b8-43a2-8aca-bfc6d6e5e2d5" containerName="payload-processing" Oct 30 22:28:11.903325 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.903304 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:11.961290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:11.961251 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:28:12.011634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.007883 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b24ff0\"" Oct 30 22:28:12.011634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.008022 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-93195d81-dockercfg-x8bfm\"" Oct 30 22:28:12.016382 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.016332 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.016571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.016484 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqnq\" (UniqueName: \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.057742 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.057708 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:28:12.061172 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.061153 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.116133 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.116102 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:28:12.117684 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.117655 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqnq\" (UniqueName: \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.117831 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.117768 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.118660 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.118599 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.179390 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.179312 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqnq\" (UniqueName: \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\") pod \"payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.216001 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.215975 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:12.218359 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.218333 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx988\" (UniqueName: \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.218490 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.218476 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.319672 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.319635 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.319934 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.319724 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx988\" (UniqueName: \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.320798 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.320767 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.381601 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.381563 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx988\" (UniqueName: \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\") pod \"payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:12.588863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.588838 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:28:12.591388 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:28:12.591360 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded013135_6f49_4a21_a634_6fb90f6c87ce.slice/crio-dabfaf0ada80861aedfa244ecba7a8168ab48dff7c94c8ef583c781f93ff1761 WatchSource:0}: Error finding container dabfaf0ada80861aedfa244ecba7a8168ab48dff7c94c8ef583c781f93ff1761: Status 404 returned error can't find the container with id dabfaf0ada80861aedfa244ecba7a8168ab48dff7c94c8ef583c781f93ff1761 Oct 30 22:28:12.674197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:12.674157 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:13.056421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.056395 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:28:13.058927 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:28:13.058900 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19458be_1a1f_4358_8fcd_647c3988203f.slice/crio-3d58f7134af4fb9c5f77c6a9016f01776755db5e9648c9d5d375f5ab9be2616a WatchSource:0}: Error finding container 3d58f7134af4fb9c5f77c6a9016f01776755db5e9648c9d5d375f5ab9be2616a: Status 404 returned error can't find the container with id 3d58f7134af4fb9c5f77c6a9016f01776755db5e9648c9d5d375f5ab9be2616a Oct 30 22:28:13.212505 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.212388 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" event={"ID":"ed013135-6f49-4a21-a634-6fb90f6c87ce","Type":"ContainerStarted","Data":"7ed287b0121dac685a009da7c5fca102c90c48585c28150ad8dbdd8f97175b94"} Oct 30 22:28:13.212714 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.212530 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" event={"ID":"ed013135-6f49-4a21-a634-6fb90f6c87ce","Type":"ContainerStarted","Data":"dabfaf0ada80861aedfa244ecba7a8168ab48dff7c94c8ef583c781f93ff1761"} Oct 30 22:28:13.212714 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.212562 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:13.214062 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.214036 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" event={"ID":"a19458be-1a1f-4358-8fcd-647c3988203f","Type":"ContainerStarted","Data":"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05"} Oct 30 22:28:13.214217 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.214082 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" event={"ID":"a19458be-1a1f-4358-8fcd-647c3988203f","Type":"ContainerStarted","Data":"3d58f7134af4fb9c5f77c6a9016f01776755db5e9648c9d5d375f5ab9be2616a"} Oct 30 22:28:13.214217 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.214130 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:13.329333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.329232 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" podStartSLOduration=2.32921818 podStartE2EDuration="2.32921818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:13.32786049 +0000 UTC m=+3918.001907026" watchObservedRunningTime="2026-10-30 22:28:13.32921818 +0000 UTC m=+3918.003264716" Oct 30 22:28:13.459729 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:13.457824 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" podStartSLOduration=1.457801136 podStartE2EDuration="1.457801136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:13.456329519 +0000 UTC m=+3918.130376072" watchObservedRunningTime="2026-10-30 22:28:13.457801136 +0000 UTC m=+3918.131847677" Oct 30 22:28:14.507566 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.507535 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:28:14.510990 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.510972 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.563518 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.563487 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b24ff0-serving-cert\"" Oct 30 22:28:14.616585 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.616552 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:28:14.643129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.643096 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.643333 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.643162 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.643410 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.643327 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.643410 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.643379 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bs6\" (UniqueName: \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.744027 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.744118 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.744192 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.744241 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bs6\" (UniqueName: \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.746150 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:28:14.746550 4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-serving-cert: secret "maas-api-e2e-worker-single-main-b24ff0-serving-cert" not found Oct 30 22:28:14.751675 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:28:14.746657 4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls podName:0ba233fd-298a-44cc-aaa5-1de1011ca291 nodeName:}" failed. No retries permitted until 2026-10-30 22:28:15.246637693 +0000 UTC m=+3919.920684211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls") pod "maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" (UID: "0ba233fd-298a-44cc-aaa5-1de1011ca291") : secret "maas-api-e2e-worker-single-main-b24ff0-serving-cert" not found Oct 30 22:28:14.755493 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.755463 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:14.809160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:14.809090 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bs6\" (UniqueName: \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:15.249827 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:15.249794 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:15.252357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:15.252330 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:15.423669 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:15.423632 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:15.797127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:15.797085 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:28:15.799349 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:28:15.799320 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba233fd_298a_44cc_aaa5_1de1011ca291.slice/crio-6a2e85a978d0800e234b1a8ee91cf340274a3ebcd193c9c81d94a9f056695679 WatchSource:0}: Error finding container 6a2e85a978d0800e234b1a8ee91cf340274a3ebcd193c9c81d94a9f056695679: Status 404 returned error can't find the container with id 6a2e85a978d0800e234b1a8ee91cf340274a3ebcd193c9c81d94a9f056695679 Oct 30 22:28:15.800959 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:15.800938 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:28:16.228588 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:16.228551 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" event={"ID":"0ba233fd-298a-44cc-aaa5-1de1011ca291","Type":"ContainerStarted","Data":"6a2e85a978d0800e234b1a8ee91cf340274a3ebcd193c9c81d94a9f056695679"} Oct 30 22:28:17.236442 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:17.236391 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" event={"ID":"0ba233fd-298a-44cc-aaa5-1de1011ca291","Type":"ContainerStarted","Data":"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed"} Oct 30 22:28:17.236976 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:17.236493 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:17.356513 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:17.356450 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" podStartSLOduration=2.484306386 podStartE2EDuration="3.356436496s" totalImagesPullingTime="872.13011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:17.35506168 +0000 UTC m=+3922.029108218" watchObservedRunningTime="2026-10-30 22:28:17.356436496 +0000 UTC m=+3922.030483032" Oct 30 22:28:22.759651 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:22.759591 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:28:23.246688 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:23.246658 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:28:24.220887 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:24.220862 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:28:24.221338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:24.220923 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:28:35.395739 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:28:35.395701 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:28:47.437839 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.437811 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:28:47.441362 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.441347 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.544735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.544704 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b24ff0\"/\"openshift-service-ca.crt\"" Oct 30 22:28:47.544735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.544724 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b24ff0\"/\"simulator-single-main-b24ff0-kserve-self-signed-certs\"" Oct 30 22:28:47.544735 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.544705 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b24ff0\"/\"kube-root-ca.crt\"" Oct 30 22:28:47.544991 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.544866 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b24ff0\"/\"default-dockercfg-chpwc\"" Oct 30 22:28:47.546846 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.546826 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:28:47.590478 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.590433 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfsw\" (UniqueName: \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.594541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.590495 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.594541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.590636 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.594541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.592691 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.594541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.592772 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.594541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.592890 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694111 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694019 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfsw\" (UniqueName: \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694111 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694062 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694111 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694087 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694169 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694218 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694264 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694524 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.694726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.694575 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.695643 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.695220 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.696918 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.696899 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.697285 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.697269 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.756303 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.756267 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfsw\" (UniqueName: \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\") pod \"simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:47.763556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:47.763517 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:48.185463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:48.185434 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:28:48.187681 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:28:48.187653 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7cc93b4_7a78_4b90_91f1_c80c6dcc9374.slice/crio-0529d5823e534a70ffebb932d11e1fbdfd5095928530cf76e046f292d9a50e48 WatchSource:0}: Error finding container 0529d5823e534a70ffebb932d11e1fbdfd5095928530cf76e046f292d9a50e48: Status 404 returned error can't find the container with id 0529d5823e534a70ffebb932d11e1fbdfd5095928530cf76e046f292d9a50e48 Oct 30 22:28:48.380767 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:48.380731 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" event={"ID":"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374","Type":"ContainerStarted","Data":"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584"} Oct 30 22:28:48.380767 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:48.380771 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" event={"ID":"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374","Type":"ContainerStarted","Data":"0529d5823e534a70ffebb932d11e1fbdfd5095928530cf76e046f292d9a50e48"} Oct 30 22:28:48.508157 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:48.508050 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" podStartSLOduration=1.507993435 podStartE2EDuration="1.507993435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:48.505802584 +0000 UTC m=+3953.179849120" watchObservedRunningTime="2026-10-30 22:28:48.507993435 +0000 UTC m=+3953.182039971" Oct 30 22:28:48.763752 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:48.763668 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:52.269290 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:52.269232 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:28:57.764421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:57.764389 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:57.777323 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:57.777293 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:28:58.436030 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:28:58.435996 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:29:05.395133 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:29:05.395090 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:29:27.347707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:29:27.347670 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:29:35.395682 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:29:35.395643 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:29:55.462476 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:29:55.462433 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:29:58.486929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:29:58.486890 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:30:05.396041 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:30:05.395998 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:30:05.606582 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:05.606547 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:30:05.716863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:05.716752 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:30:10.607726 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:10.607686 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" containerID="cri-o://419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05" gracePeriod=25 Oct 30 22:30:10.717741 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:10.717701 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" containerID="cri-o://7ed287b0121dac685a009da7c5fca102c90c48585c28150ad8dbdd8f97175b94" gracePeriod=25 Oct 30 22:30:12.948168 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:12.948134 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:30:12.948685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:12.948318 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerName="maas-api" containerID="cri-o://72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed" gracePeriod=30 Oct 30 22:30:13.242357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.242268 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.176:8443/health\": dial tcp 10.133.0.176:8443: connect: connection refused" Oct 30 22:30:13.507360 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.507263 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" podUID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" containerName="main" containerID="cri-o://1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584" gracePeriod=45 Oct 30 22:30:13.768924 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.768859 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:30:13.785719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.785690 4525 generic.go:356] "Generic (PLEG): container finished" podID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" containerID="1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584" exitCode=0 Oct 30 22:30:13.785875 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.785730 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" Oct 30 22:30:13.785875 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.785771 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" event={"ID":"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374","Type":"ContainerDied","Data":"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584"} Oct 30 22:30:13.785875 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.785820 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4" event={"ID":"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374","Type":"ContainerDied","Data":"0529d5823e534a70ffebb932d11e1fbdfd5095928530cf76e046f292d9a50e48"} Oct 30 22:30:13.785875 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.785843 4525 scope.go:122] "RemoveContainer" containerID="1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584" Oct 30 22:30:13.794147 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.794131 4525 scope.go:122] "RemoveContainer" containerID="1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584" Oct 30 22:30:13.794407 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:30:13.794389 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584\": container with ID starting with 1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584 not found: ID does not exist" containerID="1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584" Oct 30 22:30:13.794466 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.794417 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584"} err="failed to get container status \"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584\": rpc error: code = NotFound desc = could not find container \"1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584\": container with ID starting with 1f19366420fc60a5ee6e71124c44597a59b70b51c439031fa96526969e92d584 not found: ID does not exist" Oct 30 22:30:13.889668 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889637 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.889833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889741 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\" (UniqueName: \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.889833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889816 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.889919 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889867 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.889919 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889903 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.890023 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.889949 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\") pod \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\" (UID: \"b7cc93b4-7a78-4b90-91f1-c80c6dcc9374\") " Oct 30 22:30:13.890142 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890101 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:13.890218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890161 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:13.890218 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890174 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:13.890339 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890322 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.890398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890346 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.890398 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.890362 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.891985 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.891962 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:13.892095 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.892013 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "kube-api-access-pnfsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:13.892095 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.892065 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs" pod "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" (UID: "b7cc93b4-7a78-4b90-91f1-c80c6dcc9374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:13.990953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.990920 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.990953 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.990950 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:13.991365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:13.990962 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfsw\" (UniqueName: \"kubernetes.io/projected/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374-kube-api-access-pnfsw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:14.193389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.193367 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:30:14.211074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.211048 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:30:14.213490 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.213466 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b24ff0/simulator-single-main-b24ff0-kserve-7c774845c8-xqkt4"] Oct 30 22:30:14.219131 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.219103 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:14.219248 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.219107 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:14.293731 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.293644 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\" (UniqueName: \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\") pod \"0ba233fd-298a-44cc-aaa5-1de1011ca291\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " Oct 30 22:30:14.293731 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.293703 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\") pod \"0ba233fd-298a-44cc-aaa5-1de1011ca291\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " Oct 30 22:30:14.293982 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.293736 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") pod \"0ba233fd-298a-44cc-aaa5-1de1011ca291\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " Oct 30 22:30:14.293982 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.293924 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\") pod \"0ba233fd-298a-44cc-aaa5-1de1011ca291\" (UID: \"0ba233fd-298a-44cc-aaa5-1de1011ca291\") " Oct 30 22:30:14.294338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.294308 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca" pod "0ba233fd-298a-44cc-aaa5-1de1011ca291" (UID: "0ba233fd-298a-44cc-aaa5-1de1011ca291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:14.296095 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.296066 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls" pod "0ba233fd-298a-44cc-aaa5-1de1011ca291" (UID: "0ba233fd-298a-44cc-aaa5-1de1011ca291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:14.296095 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.296087 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6" pod "0ba233fd-298a-44cc-aaa5-1de1011ca291" (UID: "0ba233fd-298a-44cc-aaa5-1de1011ca291"). InnerVolumeSpecName "kube-api-access-99bs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:14.296251 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.296092 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs" pod "0ba233fd-298a-44cc-aaa5-1de1011ca291" (UID: "0ba233fd-298a-44cc-aaa5-1de1011ca291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:14.395175 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.395144 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99bs6\" (UniqueName: \"kubernetes.io/projected/0ba233fd-298a-44cc-aaa5-1de1011ca291-kube-api-access-99bs6\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:14.395175 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.395174 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:14.395175 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.395184 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba233fd-298a-44cc-aaa5-1de1011ca291-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:14.395423 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.395192 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba233fd-298a-44cc-aaa5-1de1011ca291-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:14.791399 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.791368 4525 generic.go:356] "Generic (PLEG): container finished" podID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerID="72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed" exitCode=0 Oct 30 22:30:14.791593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.791411 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" Oct 30 22:30:14.791593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.791453 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" event={"ID":"0ba233fd-298a-44cc-aaa5-1de1011ca291","Type":"ContainerDied","Data":"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed"} Oct 30 22:30:14.791593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.791510 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277" event={"ID":"0ba233fd-298a-44cc-aaa5-1de1011ca291","Type":"ContainerDied","Data":"6a2e85a978d0800e234b1a8ee91cf340274a3ebcd193c9c81d94a9f056695679"} Oct 30 22:30:14.791593 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.791540 4525 scope.go:122] "RemoveContainer" containerID="72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed" Oct 30 22:30:14.801324 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.801306 4525 scope.go:122] "RemoveContainer" containerID="72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed" Oct 30 22:30:14.801597 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:30:14.801578 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed\": container with ID starting with 72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed not found: ID does not exist" containerID="72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed" Oct 30 22:30:14.801666 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.801619 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed"} err="failed to get container status \"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed\": rpc error: code = NotFound desc = could not find container \"72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed\": container with ID starting with 72d9f105ebff6211512b8c441b887fd9545f11138a05b6e1eb1c69b15693c6ed not found: ID does not exist" Oct 30 22:30:14.956367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.956332 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:30:14.962299 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:14.962271 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b24ff0-654f94cdb-gn277"] Oct 30 22:30:15.479310 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:15.479276 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" path="/var/lib/kubelet/pods/0ba233fd-298a-44cc-aaa5-1de1011ca291/volumes" Oct 30 22:30:15.479732 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:15.479697 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" path="/var/lib/kubelet/pods/b7cc93b4-7a78-4b90-91f1-c80c6dcc9374/volumes" Oct 30 22:30:15.982328 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:15.982286 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:30:24.219355 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:24.219307 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:24.219809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:24.219422 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:30.866138 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.866059 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:30:30.867794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.867768 4525 generic.go:356] "Generic (PLEG): container finished" podID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerID="7ed287b0121dac685a009da7c5fca102c90c48585c28150ad8dbdd8f97175b94" exitCode=0 Oct 30 22:30:30.867940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.867846 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" event={"ID":"ed013135-6f49-4a21-a634-6fb90f6c87ce","Type":"ContainerDied","Data":"7ed287b0121dac685a009da7c5fca102c90c48585c28150ad8dbdd8f97175b94"} Oct 30 22:30:30.869754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.869196 4525 generic.go:356] "Generic (PLEG): container finished" podID="a19458be-1a1f-4358-8fcd-647c3988203f" containerID="419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05" exitCode=0 Oct 30 22:30:30.869754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.869282 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" event={"ID":"a19458be-1a1f-4358-8fcd-647c3988203f","Type":"ContainerDied","Data":"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05"} Oct 30 22:30:30.869754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.869324 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" event={"ID":"a19458be-1a1f-4358-8fcd-647c3988203f","Type":"ContainerDied","Data":"3d58f7134af4fb9c5f77c6a9016f01776755db5e9648c9d5d375f5ab9be2616a"} Oct 30 22:30:30.869754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.869355 4525 scope.go:122] "RemoveContainer" containerID="419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05" Oct 30 22:30:30.869754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.869507 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw" Oct 30 22:30:30.879087 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.879068 4525 scope.go:122] "RemoveContainer" containerID="419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05" Oct 30 22:30:30.879519 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:30:30.879498 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05\": container with ID starting with 419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05 not found: ID does not exist" containerID="419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05" Oct 30 22:30:30.879636 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.879528 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05"} err="failed to get container status \"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05\": rpc error: code = NotFound desc = could not find container \"419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05\": container with ID starting with 419a6652882e696ffb820550db3f736046538fc5477161b2b4470c2e8cc7ea05 not found: ID does not exist" Oct 30 22:30:30.956130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.956101 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\" (UniqueName: \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\") pod \"a19458be-1a1f-4358-8fcd-647c3988203f\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " Oct 30 22:30:30.956463 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.956177 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\") pod \"a19458be-1a1f-4358-8fcd-647c3988203f\" (UID: \"a19458be-1a1f-4358-8fcd-647c3988203f\") " Oct 30 22:30:30.956793 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.956763 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume" pod "a19458be-1a1f-4358-8fcd-647c3988203f" (UID: "a19458be-1a1f-4358-8fcd-647c3988203f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:30.958652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.958591 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988" pod "a19458be-1a1f-4358-8fcd-647c3988203f" (UID: "a19458be-1a1f-4358-8fcd-647c3988203f"). InnerVolumeSpecName "kube-api-access-vx988". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:30.977541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:30.977512 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:30:31.057596 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.057559 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx988\" (UniqueName: \"kubernetes.io/projected/a19458be-1a1f-4358-8fcd-647c3988203f-kube-api-access-vx988\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:31.057596 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.057590 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19458be-1a1f-4358-8fcd-647c3988203f-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:31.158584 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.158550 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\") pod \"ed013135-6f49-4a21-a634-6fb90f6c87ce\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " Oct 30 22:30:31.158798 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.158642 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\" (UniqueName: \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\") pod \"ed013135-6f49-4a21-a634-6fb90f6c87ce\" (UID: \"ed013135-6f49-4a21-a634-6fb90f6c87ce\") " Oct 30 22:30:31.158995 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.158968 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume" pod "ed013135-6f49-4a21-a634-6fb90f6c87ce" (UID: "ed013135-6f49-4a21-a634-6fb90f6c87ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:31.161114 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.161089 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq" pod "ed013135-6f49-4a21-a634-6fb90f6c87ce" (UID: "ed013135-6f49-4a21-a634-6fb90f6c87ce"). InnerVolumeSpecName "kube-api-access-rzqnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:31.260073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.260043 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed013135-6f49-4a21-a634-6fb90f6c87ce-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:31.260073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.260068 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzqnq\" (UniqueName: \"kubernetes.io/projected/ed013135-6f49-4a21-a634-6fb90f6c87ce-kube-api-access-rzqnq\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:31.290332 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.290300 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:30:31.294447 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.294417 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b24ff0-7679b9cd9k9jkw"] Oct 30 22:30:31.471921 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.471840 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" path="/var/lib/kubelet/pods/a19458be-1a1f-4358-8fcd-647c3988203f/volumes" Oct 30 22:30:31.875397 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.875373 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" Oct 30 22:30:31.875979 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.875405 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h" event={"ID":"ed013135-6f49-4a21-a634-6fb90f6c87ce","Type":"ContainerDied","Data":"dabfaf0ada80861aedfa244ecba7a8168ab48dff7c94c8ef583c781f93ff1761"} Oct 30 22:30:31.875979 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:31.875477 4525 scope.go:122] "RemoveContainer" containerID="7ed287b0121dac685a009da7c5fca102c90c48585c28150ad8dbdd8f97175b94" Oct 30 22:30:32.040621 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:32.040570 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:30:32.090584 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:32.090551 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b24ff0-6695ffwd4h"] Oct 30 22:30:33.472304 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:33.472272 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" path="/var/lib/kubelet/pods/ed013135-6f49-4a21-a634-6fb90f6c87ce/volumes" Oct 30 22:30:35.397102 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:30:35.397069 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:30:56.600157 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600105 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600489 4525 cpu_manager.go:414] "containerMap: removing container" podUID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" containerName="main" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600501 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" containerName="main" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600519 4525 cpu_manager.go:414] "containerMap: removing container" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600525 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600538 4525 cpu_manager.go:414] "containerMap: removing container" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600543 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600553 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerName="maas-api" Oct 30 22:30:56.600558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600558 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerName="maas-api" Oct 30 22:30:56.600837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600624 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cc93b4-7a78-4b90-91f1-c80c6dcc9374" containerName="main" Oct 30 22:30:56.600837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600635 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19458be-1a1f-4358-8fcd-647c3988203f" containerName="payload-processing" Oct 30 22:30:56.600837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600643 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed013135-6f49-4a21-a634-6fb90f6c87ce" containerName="payload-pre-processing" Oct 30 22:30:56.600837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.600648 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba233fd-298a-44cc-aaa5-1de1011ca291" containerName="maas-api" Oct 30 22:30:56.603664 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.603644 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683231 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683283 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683384 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683411 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-959qw\" (UniqueName: \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683464 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.689631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.683499 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.724798 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.724765 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eb49cee1-kserve-self-signed-certs\"" Oct 30 22:30:56.726783 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.726762 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:30:56.784795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.784763 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.784795 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.784798 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.784904 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.784930 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959qw\" (UniqueName: \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785022 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.784976 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785197 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.785165 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785377 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.785357 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.785357 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.785458 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.785432 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.787444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.787419 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.787967 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.787943 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.851129 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.851055 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-959qw\" (UniqueName: \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\") pod \"e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:56.932978 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:56.932940 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:30:57.319477 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:57.319453 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:30:57.321153 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:30:57.321133 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01a18d8d_3916_4599_beac_100c2b7421dc.slice/crio-f47ba45efe62e7eed6af3be777f942e9a7d6bd9f83524e2bc356aad0ce70498c WatchSource:0}: Error finding container f47ba45efe62e7eed6af3be777f942e9a7d6bd9f83524e2bc356aad0ce70498c: Status 404 returned error can't find the container with id f47ba45efe62e7eed6af3be777f942e9a7d6bd9f83524e2bc356aad0ce70498c Oct 30 22:30:58.002956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:58.002866 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" event={"ID":"01a18d8d-3916-4599-beac-100c2b7421dc","Type":"ContainerStarted","Data":"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5"} Oct 30 22:30:58.002956 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:58.002909 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" event={"ID":"01a18d8d-3916-4599-beac-100c2b7421dc","Type":"ContainerStarted","Data":"f47ba45efe62e7eed6af3be777f942e9a7d6bd9f83524e2bc356aad0ce70498c"} Oct 30 22:30:58.121545 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:58.121491 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" podStartSLOduration=2.121476647 podStartE2EDuration="2.121476647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:58.119762251 +0000 UTC m=+4082.793808788" watchObservedRunningTime="2026-10-30 22:30:58.121476647 +0000 UTC m=+4082.795523240" Oct 30 22:30:58.934070 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:30:58.934033 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:31:01.223438 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:01.223402 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:31:05.395022 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:31:05.394975 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:31:06.933998 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:06.933921 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:31:06.946443 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:06.946417 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:31:07.053374 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:07.053337 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:31:26.076851 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:26.076810 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:31:35.395833 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:31:35.395793 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:31:57.357781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:31:57.357748 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:32:05.396125 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:32:05.396090 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:32:08.868838 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:08.868803 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:32:12.945009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:12.944965 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:32:23.889484 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:23.889419 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" containerName="main" containerID="cri-o://fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5" gracePeriod=45 Oct 30 22:32:24.043272 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.043234 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.178:8000/ready\": dial tcp 10.133.0.178:8000: connect: connection refused" Oct 30 22:32:24.142895 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.142827 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:32:24.181090 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181061 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181113 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181149 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181171 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\" (UniqueName: \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181216 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181265 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\") pod \"01a18d8d-3916-4599-beac-100c2b7421dc\" (UID: \"01a18d8d-3916-4599-beac-100c2b7421dc\") " Oct 30 22:32:24.181491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181375 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:24.181491 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181428 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:24.181679 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181623 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:24.181739 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181705 4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-model-cache\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.181739 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.181720 4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-home\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.183641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.183586 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:24.183641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.183626 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "kube-api-access-959qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:24.183825 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.183652 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm" pod "01a18d8d-3916-4599-beac-100c2b7421dc" (UID: "01a18d8d-3916-4599-beac-100c2b7421dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:24.283326 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.283291 4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-dshm\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.283480 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.283332 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-959qw\" (UniqueName: \"kubernetes.io/projected/01a18d8d-3916-4599-beac-100c2b7421dc-kube-api-access-959qw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.283480 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.283349 4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a18d8d-3916-4599-beac-100c2b7421dc-tls-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.283480 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.283364 4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a18d8d-3916-4599-beac-100c2b7421dc-tmp-dir\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:24.444444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.444359 4525 generic.go:356] "Generic (PLEG): container finished" podID="01a18d8d-3916-4599-beac-100c2b7421dc" containerID="fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5" exitCode=0 Oct 30 22:32:24.444444 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.444402 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" Oct 30 22:32:24.444716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.444441 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" event={"ID":"01a18d8d-3916-4599-beac-100c2b7421dc","Type":"ContainerDied","Data":"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5"} Oct 30 22:32:24.444716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.444485 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm" event={"ID":"01a18d8d-3916-4599-beac-100c2b7421dc","Type":"ContainerDied","Data":"f47ba45efe62e7eed6af3be777f942e9a7d6bd9f83524e2bc356aad0ce70498c"} Oct 30 22:32:24.444716 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.444508 4525 scope.go:122] "RemoveContainer" containerID="fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5" Oct 30 22:32:24.453812 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.453795 4525 scope.go:122] "RemoveContainer" containerID="fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5" Oct 30 22:32:24.454084 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:32:24.454068 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5\": container with ID starting with fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5 not found: ID does not exist" containerID="fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5" Oct 30 22:32:24.454123 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.454093 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5"} err="failed to get container status \"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5\": rpc error: code = NotFound desc = could not find container \"fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5\": container with ID starting with fb031bf24873435b3b8535964a44cbf04c45501c617e8fb7782cfbd05796afb5 not found: ID does not exist" Oct 30 22:32:24.610652 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.610599 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:32:24.613492 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.613467 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eb49cee1-kserve-66d9ddc7fc-zp6sm"] Oct 30 22:32:24.619250 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:24.619227 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:32:25.475000 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:25.474961 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" path="/var/lib/kubelet/pods/01a18d8d-3916-4599-beac-100c2b7421dc/volumes" Oct 30 22:32:35.395840 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:32:35.395797 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:32:36.946849 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:36.946765 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:32:45.474305 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:32:45.474271 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:33:05.397104 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:33:05.397062 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:33:06.537718 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:06.537676 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:33:07.036667 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.036630 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm"] Oct 30 22:33:07.037111 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.037092 4525 cpu_manager.go:414] "containerMap: removing container" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" containerName="main" Oct 30 22:33:07.037200 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.037113 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" containerName="main" Oct 30 22:33:07.037260 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.037207 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a18d8d-3916-4599-beac-100c2b7421dc" containerName="main" Oct 30 22:33:07.040361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.040340 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.136920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.136882 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-metrics-certs\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.137085 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.136995 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0e822-6aaa-4529-bc45-d62246ca639f-openshift-service-ca\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.137085 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.137029 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-maas-api-tls\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.137181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.137105 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/aaf0e822-6aaa-4529-bc45-d62246ca639f-kube-api-access-bwd4j\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.145903 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.145877 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm"] Oct 30 22:33:07.238399 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.238359 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/aaf0e822-6aaa-4529-bc45-d62246ca639f-kube-api-access-bwd4j\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.238583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.238471 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-metrics-certs\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.238583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.238504 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0e822-6aaa-4529-bc45-d62246ca639f-openshift-service-ca\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.238583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.238519 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-maas-api-tls\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.239294 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.239270 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf0e822-6aaa-4529-bc45-d62246ca639f-openshift-service-ca\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.241361 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.241338 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-metrics-certs\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.241432 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.241338 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaf0e822-6aaa-4529-bc45-d62246ca639f-maas-api-tls\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.299896 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.299832 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwd4j\" (UniqueName: \"kubernetes.io/projected/aaf0e822-6aaa-4529-bc45-d62246ca639f-kube-api-access-bwd4j\") pod \"maas-api-5bb5fc9b7-kbvzm\" (UID: \"aaf0e822-6aaa-4529-bc45-d62246ca639f\") " pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.353798 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.353768 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:07.738706 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:07.738682 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm"] Oct 30 22:33:07.741683 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:33:07.741654 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf0e822_6aaa_4529_bc45_d62246ca639f.slice/crio-0ced975e2f58f53b73e1ebba142b8be2c92909e90f954e1719d18588f6772ca5 WatchSource:0}: Error finding container 0ced975e2f58f53b73e1ebba142b8be2c92909e90f954e1719d18588f6772ca5: Status 404 returned error can't find the container with id 0ced975e2f58f53b73e1ebba142b8be2c92909e90f954e1719d18588f6772ca5 Oct 30 22:33:08.704996 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:08.704959 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" event={"ID":"aaf0e822-6aaa-4529-bc45-d62246ca639f","Type":"ContainerStarted","Data":"dfa8c134d350c9d587f75a67462d3061b4261fcbb70a54ae60a5c28d815068b9"} Oct 30 22:33:08.705243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:08.705025 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" event={"ID":"aaf0e822-6aaa-4529-bc45-d62246ca639f","Type":"ContainerStarted","Data":"0ced975e2f58f53b73e1ebba142b8be2c92909e90f954e1719d18588f6772ca5"} Oct 30 22:33:08.705243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:08.705053 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:08.827707 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:08.827555 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" podStartSLOduration=2.021201029 podStartE2EDuration="2.827535547s" totalImagesPullingTime="806.334518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:08.827215235 +0000 UTC m=+4213.501261769" watchObservedRunningTime="2026-10-30 22:33:08.827535547 +0000 UTC m=+4213.501582084" Oct 30 22:33:14.279079 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.279044 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:33:14.282700 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.282682 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.335446 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.335417 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9871e8fb-openshift-default-dockercfg-dw758\"" Oct 30 22:33:14.388063 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.388024 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:33:14.418078 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418037 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418078 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418090 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418173 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418200 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418298 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418326 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418363 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418354 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418397 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.418704 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.418441 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.518902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.518862 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.518902 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.518903 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.518925 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.518964 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519094 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519153 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519236 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519178 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519260 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519291 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519401 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519508 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519433 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519582 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.519754 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.519688 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.520043 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.520002 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.522063 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.522040 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.522272 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.522257 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.583619 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.583531 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.588545 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.588518 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\") pod \"e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.595885 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.595855 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:14.711468 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:14.711431 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" podUID="aaf0e822-6aaa-4529-bc45-d62246ca639f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.179:8443/health\": dial tcp 10.133.0.179:8443: connect: connection refused" Oct 30 22:33:15.023814 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.023750 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:33:15.026100 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:33:15.026072 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0282b91c_1648_4578_b4f3_c81f43b049ed.slice/crio-8b70d198ab2d3a86b8614515bffa5c2df003f9bc019676ec99ecce6cbc7e343d WatchSource:0}: Error finding container 8b70d198ab2d3a86b8614515bffa5c2df003f9bc019676ec99ecce6cbc7e343d: Status 404 returned error can't find the container with id 8b70d198ab2d3a86b8614515bffa5c2df003f9bc019676ec99ecce6cbc7e343d Oct 30 22:33:15.028143 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.028111 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:15.028212 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.028182 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:15.028249 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.028211 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:15.740871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.740835 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerStarted","Data":"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc"} Oct 30 22:33:15.741266 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.740897 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerStarted","Data":"8b70d198ab2d3a86b8614515bffa5c2df003f9bc019676ec99ecce6cbc7e343d"} Oct 30 22:33:15.863946 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:15.863871 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podStartSLOduration=1.8638491560000001 podStartE2EDuration="1.863849156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:15.862253681 +0000 UTC m=+4220.536300217" watchObservedRunningTime="2026-10-30 22:33:15.863849156 +0000 UTC m=+4220.537895692" Oct 30 22:33:16.596557 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:16.596516 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:16.601698 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:16.601673 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:16.746167 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:16.746140 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:16.747162 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:16.747145 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:18.756929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:18.756892 4525 generic.go:356] "Generic (PLEG): container finished" podID="aaf0e822-6aaa-4529-bc45-d62246ca639f" containerID="dfa8c134d350c9d587f75a67462d3061b4261fcbb70a54ae60a5c28d815068b9" exitCode=1 Oct 30 22:33:18.757353 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:18.756966 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" event={"ID":"aaf0e822-6aaa-4529-bc45-d62246ca639f","Type":"ContainerDied","Data":"dfa8c134d350c9d587f75a67462d3061b4261fcbb70a54ae60a5c28d815068b9"} Oct 30 22:33:18.757429 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:18.757417 4525 scope.go:122] "RemoveContainer" containerID="dfa8c134d350c9d587f75a67462d3061b4261fcbb70a54ae60a5c28d815068b9" Oct 30 22:33:18.757789 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:18.757775 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:33:19.763920 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:19.763887 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" event={"ID":"aaf0e822-6aaa-4529-bc45-d62246ca639f","Type":"ContainerStarted","Data":"a159ff8ca2ac512914b3fa0d7aa04a48dbabae7448a3cd40d00ad9cf5f26cc66"} Oct 30 22:33:19.764357 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:19.764100 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:25.774249 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:25.774216 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bb5fc9b7-kbvzm" Oct 30 22:33:25.944495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:25.944448 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 22:33:25.944750 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:25.944721 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" podUID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" containerName="maas-api" containerID="cri-o://350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466" gracePeriod=30 Oct 30 22:33:27.288861 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.288835 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 22:33:27.352998 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.352910 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\" (UniqueName: \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\") pod \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " Oct 30 22:33:27.352998 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.352956 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\") pod \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " Oct 30 22:33:27.353246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.353103 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\") pod \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " Oct 30 22:33:27.353246 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.353136 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\") pod \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\" (UID: \"014bcc65-dedd-4cdb-8c95-210fdc24ffe7\") " Oct 30 22:33:27.353426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.353399 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca" pod "014bcc65-dedd-4cdb-8c95-210fdc24ffe7" (UID: "014bcc65-dedd-4cdb-8c95-210fdc24ffe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:27.353512 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.353478 4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-openshift-service-ca\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.355424 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.355397 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs" pod "014bcc65-dedd-4cdb-8c95-210fdc24ffe7" (UID: "014bcc65-dedd-4cdb-8c95-210fdc24ffe7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:27.355555 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.355458 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls" pod "014bcc65-dedd-4cdb-8c95-210fdc24ffe7" (UID: "014bcc65-dedd-4cdb-8c95-210fdc24ffe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:27.355639 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.355563 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg" pod "014bcc65-dedd-4cdb-8c95-210fdc24ffe7" (UID: "014bcc65-dedd-4cdb-8c95-210fdc24ffe7"). InnerVolumeSpecName "kube-api-access-5wjbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:27.454013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.453980 4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-metrics-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.454013 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.454011 4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-maas-api-tls\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.454233 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.454024 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjbg\" (UniqueName: \"kubernetes.io/projected/014bcc65-dedd-4cdb-8c95-210fdc24ffe7-kube-api-access-5wjbg\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:33:27.803560 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.803504 4525 generic.go:356] "Generic (PLEG): container finished" podID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" containerID="350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466" exitCode=0 Oct 30 22:33:27.804019 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.803975 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" event={"ID":"014bcc65-dedd-4cdb-8c95-210fdc24ffe7","Type":"ContainerDied","Data":"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466"} Oct 30 22:33:27.804228 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.804206 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" event={"ID":"014bcc65-dedd-4cdb-8c95-210fdc24ffe7","Type":"ContainerDied","Data":"30f85a4b1e4951b9cd33538e7a4a2868baa23d45146a591d53e618284700e99e"} Oct 30 22:33:27.804367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.804353 4525 scope.go:122] "RemoveContainer" containerID="350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466" Oct 30 22:33:27.804650 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.804634 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd" Oct 30 22:33:27.828012 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.827980 4525 scope.go:122] "RemoveContainer" containerID="350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466" Oct 30 22:33:27.828405 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:33:27.828380 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466\": container with ID starting with 350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466 not found: ID does not exist" containerID="350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466" Oct 30 22:33:27.828520 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.828415 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466"} err="failed to get container status \"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466\": rpc error: code = NotFound desc = could not find container \"350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466\": container with ID starting with 350c41834ac24f1e8ba2c092c824bd09ed646ebe6ebf0cfc41c9954741a24466 not found: ID does not exist" Oct 30 22:33:27.970871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:27.970834 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 22:33:28.020741 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:28.020703 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fd68d57f5-hqqfd"] Oct 30 22:33:29.471925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:29.471894 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" path="/var/lib/kubelet/pods/014bcc65-dedd-4cdb-8c95-210fdc24ffe7/volumes" Oct 30 22:33:30.560072 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:30.560034 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:33:35.396727 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:33:35.396685 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:33:40.887549 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.887513 4525 generic.go:356] "Generic (PLEG): container finished" podID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerID="9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc" exitCode=0 Oct 30 22:33:40.888052 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.887587 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerDied","Data":"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc"} Oct 30 22:33:40.888052 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.888002 4525 scope.go:122] "RemoveContainer" containerID="9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc" Oct 30 22:33:40.889345 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.889316 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:40.889417 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.889389 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:40.889461 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:40.889432 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:33:41.895676 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:41.895636 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerStarted","Data":"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5"} Oct 30 22:33:42.596369 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:42.596338 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:42.597682 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:42.597652 4525 patch_prober.go:29] interesting pod/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:33:42.597830 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:42.597713 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:33:43.596637 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:43.596586 4525 patch_prober.go:29] interesting pod/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:33:43.597045 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:43.596659 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:33:44.596839 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:44.596796 4525 patch_prober.go:29] interesting pod/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:33:44.597243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:44.596861 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:33:45.597128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:45.597089 4525 patch_prober.go:29] interesting pod/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:33:45.597528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:45.597155 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:33:46.596830 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:46.596798 4525 patch_prober.go:29] interesting pod/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 22:33:46.597025 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:46.596859 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 22:33:46.746073 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:46.746030 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:47.601055 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:47.601024 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:33:47.602177 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:33:47.602149 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:34:05.396348 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:34:05.396306 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:34:05.609440 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:05.609402 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:34:05.609701 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:05.609676 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" containerID="cri-o://48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5" gracePeriod=30 Oct 30 22:34:09.140194 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:09.140106 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:34:10.866310 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.866282 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:34:10.958267 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958165 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958267 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958254 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958286 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958305 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958334 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958359 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958378 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958416 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958528 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958482 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\") pod \"0282b91c-1648-4578-b4f3-c81f43b049ed\" (UID: \"0282b91c-1648-4578-b4f3-c81f43b049ed\") " Oct 30 22:34:10.958930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958582 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:10.958930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958677 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:10.958930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958737 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:10.958930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958855 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:10.958930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.958878 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:10.959176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.959013 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:10.959176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.959034 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:10.959176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.959048 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:10.959176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.959064 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0282b91c-1648-4578-b4f3-c81f43b049ed-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:10.959176 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.959093 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:10.960878 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.960839 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:10.961016 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.960891 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:10.961286 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.961262 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "kube-api-access-6lppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:10.961286 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:10.961266 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo" pod "0282b91c-1648-4578-b4f3-c81f43b049ed" (UID: "0282b91c-1648-4578-b4f3-c81f43b049ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:11.025539 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.025500 4525 generic.go:356] "Generic (PLEG): container finished" podID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerID="48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5" exitCode=0 Oct 30 22:34:11.025844 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.025549 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" Oct 30 22:34:11.025844 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.025592 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerDied","Data":"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5"} Oct 30 22:34:11.025844 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.025675 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl" event={"ID":"0282b91c-1648-4578-b4f3-c81f43b049ed","Type":"ContainerDied","Data":"8b70d198ab2d3a86b8614515bffa5c2df003f9bc019676ec99ecce6cbc7e343d"} Oct 30 22:34:11.025844 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.025701 4525 scope.go:122] "RemoveContainer" containerID="48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5" Oct 30 22:34:11.035339 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.035313 4525 scope.go:122] "RemoveContainer" containerID="9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc" Oct 30 22:34:11.044626 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.044577 4525 scope.go:122] "RemoveContainer" containerID="48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5" Oct 30 22:34:11.044974 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:34:11.044942 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5\": container with ID starting with 48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5 not found: ID does not exist" containerID="48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5" Oct 30 22:34:11.045057 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.044987 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5"} err="failed to get container status \"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5\": rpc error: code = NotFound desc = could not find container \"48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5\": container with ID starting with 48e70c195c136fb717f73007cd1c6ca2a5354bd5fad74d84d8f98110891be9a5 not found: ID does not exist" Oct 30 22:34:11.045057 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.045012 4525 scope.go:122] "RemoveContainer" containerID="9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc" Oct 30 22:34:11.047697 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:34:11.047671 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc\": container with ID starting with 9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc not found: ID does not exist" containerID="9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc" Oct 30 22:34:11.047805 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.047703 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc"} err="failed to get container status \"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc\": rpc error: code = NotFound desc = could not find container \"9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc\": container with ID starting with 9f424c7ab0cbb6e901e98b604219810e0bfd1b44270916e7f742e605ce2715bc not found: ID does not exist" Oct 30 22:34:11.060523 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.060494 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-kube-api-access-6lppp\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.060523 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.060526 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.060794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.060540 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.060794 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.060553 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0282b91c-1648-4578-b4f3-c81f43b049ed-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:11.151846 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.151808 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:34:11.156145 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.156114 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9871e8fb-openshift-default-c74d78fbf-c4tfl"] Oct 30 22:34:11.478275 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:11.478238 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" path="/var/lib/kubelet/pods/0282b91c-1648-4578-b4f3-c81f43b049ed/volumes" Oct 30 22:34:17.494368 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494329 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494722 4525 cpu_manager.go:414] "containerMap: removing container" podUID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" containerName="maas-api" Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494734 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" containerName="maas-api" Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494757 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494764 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494775 4525 cpu_manager.go:414] "containerMap: removing container" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.494821 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494780 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.495029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494831 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="014bcc65-dedd-4cdb-8c95-210fdc24ffe7" containerName="maas-api" Oct 30 22:34:17.495029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494844 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.495029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.494851 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0282b91c-1648-4578-b4f3-c81f43b049ed" containerName="istio-proxy" Oct 30 22:34:17.499541 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.499517 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521410 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521486 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521527 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521553 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521577 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521635 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521660 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521690 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzg8\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.526966 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.521755 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.553247 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.553196 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-563990-openshift-default-dockercfg-lx5qk\"" Oct 30 22:34:17.604827 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.604791 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:34:17.622503 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.622472 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.622925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.622850 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.622925 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.622908 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.622934 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.622959 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623008 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623032 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623062 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzg8\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623155 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623103 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623532 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623510 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.623850 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.623823 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.624130 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.624096 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.624338 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.624314 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.624637 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.624594 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.626127 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.626095 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.626855 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.626811 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.686294 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.686258 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.686770 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.686751 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzg8\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\") pod \"e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:17.812356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:17.812273 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:18.240303 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:18.240275 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:34:18.242243 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:34:18.242214 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b91a93_3d10_4be4_8679_f1aa0252651f.slice/crio-4b999242b2634f7c1d944638e9062e42611a5b67ea8900d6068cbccd3dcac6bc WatchSource:0}: Error finding container 4b999242b2634f7c1d944638e9062e42611a5b67ea8900d6068cbccd3dcac6bc: Status 404 returned error can't find the container with id 4b999242b2634f7c1d944638e9062e42611a5b67ea8900d6068cbccd3dcac6bc Oct 30 22:34:18.244406 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:18.244369 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:18.244514 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:18.244440 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:18.244514 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:18.244477 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:19.064500 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:19.064451 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerStarted","Data":"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f"} Oct 30 22:34:19.064922 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:19.064521 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerStarted","Data":"4b999242b2634f7c1d944638e9062e42611a5b67ea8900d6068cbccd3dcac6bc"} Oct 30 22:34:19.239337 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:19.239271 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podStartSLOduration=2.239252793 podStartE2EDuration="2.239252793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:19.236532603 +0000 UTC m=+4283.910579143" watchObservedRunningTime="2026-10-30 22:34:19.239252793 +0000 UTC m=+4283.913299334" Oct 30 22:34:19.812825 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:19.812777 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:19.818113 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:19.818089 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:20.069699 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:20.069622 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:20.070649 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:20.070623 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:25.827475 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.827436 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:34:25.831125 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.831109 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:25.923313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.922930 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x684k\" (UniqueName: \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:25.923313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.922996 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:25.935894 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.935855 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-563990-dockercfg-4l8gw\"" Oct 30 22:34:25.935894 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.935961 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-563990\"" Oct 30 22:34:25.939961 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.939931 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:34:25.994767 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.994731 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:34:25.998244 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:25.998226 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.023987 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.023960 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x684k\" (UniqueName: \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:26.024145 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.023997 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:26.024625 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.024584 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:26.056690 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.056649 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:34:26.086819 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.086740 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x684k\" (UniqueName: \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\") pod \"payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:26.124685 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.124650 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.124848 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.124820 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjv8\" (UniqueName: \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.144536 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.144508 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:26.226031 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.225997 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjv8\" (UniqueName: \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.226173 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.226053 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.226832 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.226813 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.287888 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.287860 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjv8\" (UniqueName: \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\") pod \"payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.313330 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.313306 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:26.518367 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.517873 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:34:26.520910 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:34:26.520869 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387083f2_cd41_4587_bf7c_9989dad40063.slice/crio-aaf8de4a2a214b42e60c75757d839e92e445232989da38b7282acae608cb29f4 WatchSource:0}: Error finding container aaf8de4a2a214b42e60c75757d839e92e445232989da38b7282acae608cb29f4: Status 404 returned error can't find the container with id aaf8de4a2a214b42e60c75757d839e92e445232989da38b7282acae608cb29f4 Oct 30 22:34:26.686765 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:26.686742 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:34:26.689049 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:34:26.689020 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633e1166_b903_48e0_9b0b_6454aff7745c.slice/crio-44fafdba0b0caffe1025b0e16bce61125a273ac51cf2b98c34b4e167b5f653da WatchSource:0}: Error finding container 44fafdba0b0caffe1025b0e16bce61125a273ac51cf2b98c34b4e167b5f653da: Status 404 returned error can't find the container with id 44fafdba0b0caffe1025b0e16bce61125a273ac51cf2b98c34b4e167b5f653da Oct 30 22:34:27.104054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.103958 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" event={"ID":"387083f2-cd41-4587-bf7c-9989dad40063","Type":"ContainerStarted","Data":"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921"} Oct 30 22:34:27.104054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.104028 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" event={"ID":"387083f2-cd41-4587-bf7c-9989dad40063","Type":"ContainerStarted","Data":"aaf8de4a2a214b42e60c75757d839e92e445232989da38b7282acae608cb29f4"} Oct 30 22:34:27.104054 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.104051 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:27.105499 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.105477 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" event={"ID":"633e1166-b903-48e0-9b0b-6454aff7745c","Type":"ContainerStarted","Data":"e8610fa9fe376cb3ddd5536f146556635c1ea64f1d7097db27f22fc5d6f8e1ee"} Oct 30 22:34:27.105632 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.105515 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" event={"ID":"633e1166-b903-48e0-9b0b-6454aff7745c","Type":"ContainerStarted","Data":"44fafdba0b0caffe1025b0e16bce61125a273ac51cf2b98c34b4e167b5f653da"} Oct 30 22:34:27.105632 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.105562 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:27.220916 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.220851 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" podStartSLOduration=2.220836248 podStartE2EDuration="2.220836248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:27.22024518 +0000 UTC m=+4291.894291728" watchObservedRunningTime="2026-10-30 22:34:27.220836248 +0000 UTC m=+4291.894882785" Oct 30 22:34:27.347014 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:27.346175 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" podStartSLOduration=2.346153673 podStartE2EDuration="2.346153673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:27.344819615 +0000 UTC m=+4292.018866163" watchObservedRunningTime="2026-10-30 22:34:27.346153673 +0000 UTC m=+4292.020200209" Oct 30 22:34:35.396360 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:34:35.396313 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:34:36.638424 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:36.638391 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:34:38.112969 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:38.112938 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:34:38.113366 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:38.113010 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:34:46.203189 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.203151 4525 generic.go:356] "Generic (PLEG): container finished" podID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerID="bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f" exitCode=0 Oct 30 22:34:46.203954 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.203222 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerDied","Data":"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f"} Oct 30 22:34:46.203954 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.203640 4525 scope.go:122] "RemoveContainer" containerID="bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f" Oct 30 22:34:46.204939 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.204910 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:46.205012 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.204979 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:46.205052 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:46.205011 4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:34:47.216877 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:47.216839 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerStarted","Data":"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447"} Oct 30 22:34:47.812489 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:47.812449 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:47.813857 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:47.813823 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 22:34:47.814023 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:47.813874 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 22:34:48.812980 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:48.812947 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 22:34:48.813416 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:48.813001 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 22:34:49.813549 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:49.813508 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 22:34:49.814020 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:49.813572 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 22:34:50.069945 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:50.069852 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:50.812984 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:50.812944 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 22:34:50.813216 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:50.813013 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 22:34:51.812879 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:51.812844 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 22:34:51.813404 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:51.812897 4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 22:34:52.817581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:52.817549 4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:34:52.818526 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:34:52.818507 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:35:05.396755 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:35:05.396714 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:35:35.396213 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:35:35.396174 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:36:05.396811 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:36:05.396768 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:36:29.576835 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:29.576799 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:36:29.682576 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:29.682536 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:36:34.577474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.577410 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" podUID="633e1166-b903-48e0-9b0b-6454aff7745c" containerName="payload-processing" containerID="cri-o://e8610fa9fe376cb3ddd5536f146556635c1ea64f1d7097db27f22fc5d6f8e1ee" gracePeriod=25 Oct 30 22:36:34.683727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.683675 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" podUID="387083f2-cd41-4587-bf7c-9989dad40063" containerName="payload-pre-processing" containerID="cri-o://889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921" gracePeriod=25 Oct 30 22:36:34.747021 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.746922 4525 generic.go:356] "Generic (PLEG): container finished" podID="633e1166-b903-48e0-9b0b-6454aff7745c" containerID="e8610fa9fe376cb3ddd5536f146556635c1ea64f1d7097db27f22fc5d6f8e1ee" exitCode=0 Oct 30 22:36:34.747021 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.746972 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" event={"ID":"633e1166-b903-48e0-9b0b-6454aff7745c","Type":"ContainerDied","Data":"e8610fa9fe376cb3ddd5536f146556635c1ea64f1d7097db27f22fc5d6f8e1ee"} Oct 30 22:36:34.823324 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.823299 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:36:34.862134 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.862057 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\" (UniqueName: \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\") pod \"633e1166-b903-48e0-9b0b-6454aff7745c\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " Oct 30 22:36:34.863110 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.862848 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\") pod \"633e1166-b903-48e0-9b0b-6454aff7745c\" (UID: \"633e1166-b903-48e0-9b0b-6454aff7745c\") " Oct 30 22:36:34.863558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.863251 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume" pod "633e1166-b903-48e0-9b0b-6454aff7745c" (UID: "633e1166-b903-48e0-9b0b-6454aff7745c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:34.863558 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.863525 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/633e1166-b903-48e0-9b0b-6454aff7745c-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:34.865601 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.865578 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8" pod "633e1166-b903-48e0-9b0b-6454aff7745c" (UID: "633e1166-b903-48e0-9b0b-6454aff7745c"). InnerVolumeSpecName "kube-api-access-mbjv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:34.926890 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.926862 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:36:34.964874 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.964837 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\") pod \"387083f2-cd41-4587-bf7c-9989dad40063\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " Oct 30 22:36:34.965083 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.964983 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\" (UniqueName: \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\") pod \"387083f2-cd41-4587-bf7c-9989dad40063\" (UID: \"387083f2-cd41-4587-bf7c-9989dad40063\") " Oct 30 22:36:34.965254 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.965230 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume" pod "387083f2-cd41-4587-bf7c-9989dad40063" (UID: "387083f2-cd41-4587-bf7c-9989dad40063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:34.965389 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.965368 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjv8\" (UniqueName: \"kubernetes.io/projected/633e1166-b903-48e0-9b0b-6454aff7745c-kube-api-access-mbjv8\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:34.965453 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.965394 4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387083f2-cd41-4587-bf7c-9989dad40063-plugins-config-volume\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:34.967356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:34.967332 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k" pod "387083f2-cd41-4587-bf7c-9989dad40063" (UID: "387083f2-cd41-4587-bf7c-9989dad40063"). InnerVolumeSpecName "kube-api-access-x684k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:35.066676 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.066643 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x684k\" (UniqueName: \"kubernetes.io/projected/387083f2-cd41-4587-bf7c-9989dad40063-kube-api-access-x684k\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:35.396847 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:36:35.396812 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:36:35.752587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.752500 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" Oct 30 22:36:35.752587 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.752524 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq" event={"ID":"633e1166-b903-48e0-9b0b-6454aff7745c","Type":"ContainerDied","Data":"44fafdba0b0caffe1025b0e16bce61125a273ac51cf2b98c34b4e167b5f653da"} Oct 30 22:36:35.753160 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.752594 4525 scope.go:122] "RemoveContainer" containerID="e8610fa9fe376cb3ddd5536f146556635c1ea64f1d7097db27f22fc5d6f8e1ee" Oct 30 22:36:35.754009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.753851 4525 generic.go:356] "Generic (PLEG): container finished" podID="387083f2-cd41-4587-bf7c-9989dad40063" containerID="889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921" exitCode=0 Oct 30 22:36:35.754009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.753895 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" Oct 30 22:36:35.754009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.753912 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" event={"ID":"387083f2-cd41-4587-bf7c-9989dad40063","Type":"ContainerDied","Data":"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921"} Oct 30 22:36:35.754009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.753947 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp" event={"ID":"387083f2-cd41-4587-bf7c-9989dad40063","Type":"ContainerDied","Data":"aaf8de4a2a214b42e60c75757d839e92e445232989da38b7282acae608cb29f4"} Oct 30 22:36:35.761577 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.761558 4525 scope.go:122] "RemoveContainer" containerID="889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921" Oct 30 22:36:35.769829 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.769810 4525 scope.go:122] "RemoveContainer" containerID="889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921" Oct 30 22:36:35.770130 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:36:35.770107 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921\": container with ID starting with 889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921 not found: ID does not exist" containerID="889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921" Oct 30 22:36:35.770214 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.770137 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921"} err="failed to get container status \"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921\": rpc error: code = NotFound desc = could not find container \"889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921\": container with ID starting with 889b9eaea4571c8ef858abe6883c7a67d7cb96719d843e632a2449f220a5a921 not found: ID does not exist" Oct 30 22:36:35.916730 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.916694 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:36:35.966728 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:35.966689 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-563990-6c9bfc69ff-nfvxp"] Oct 30 22:36:36.031581 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:36.031498 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:36:36.038381 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:36.038352 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-563990-5df456d87b-m46mq"] Oct 30 22:36:37.474733 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:37.474694 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387083f2-cd41-4587-bf7c-9989dad40063" path="/var/lib/kubelet/pods/387083f2-cd41-4587-bf7c-9989dad40063/volumes" Oct 30 22:36:37.475261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:37.475239 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633e1166-b903-48e0-9b0b-6454aff7745c" path="/var/lib/kubelet/pods/633e1166-b903-48e0-9b0b-6454aff7745c/volumes" Oct 30 22:36:47.950878 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:47.950840 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:36:50.974074 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:50.974039 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:36:50.974533 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:50.974255 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" containerID="cri-o://40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447" gracePeriod=30 Oct 30 22:36:52.820468 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:52.820434 4525 patch_prober.go:29] interesting pod/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:52.820909 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:52.820503 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:56.224853 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.224826 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:36:56.401393 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401360 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401578 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401418 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401578 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401462 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401578 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401538 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401578 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401574 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401654 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401686 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401729 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401784 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\") pod \"31b91a93-3d10-4be4-8679-f1aa0252651f\" (UID: \"31b91a93-3d10-4be4-8679-f1aa0252651f\") " Oct 30 22:36:56.401833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.401811 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:56.402085 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402032 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:56.402421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402158 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.402421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402369 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.402657 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402442 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:56.402657 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402531 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:56.402657 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.402648 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:56.404094 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.404070 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:56.404365 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.404331 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:56.404454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.404402 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "kube-api-access-4nzg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:56.404769 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.404750 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy" pod "31b91a93-3d10-4be4-8679-f1aa0252651f" (UID: "31b91a93-3d10-4be4-8679-f1aa0252651f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:56.503451 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503414 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503451 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503449 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503463 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b91a93-3d10-4be4-8679-f1aa0252651f-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503475 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503487 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503500 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzg8\" (UniqueName: \"kubernetes.io/projected/31b91a93-3d10-4be4-8679-f1aa0252651f-kube-api-access-4nzg8\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.503727 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.503515 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b91a93-3d10-4be4-8679-f1aa0252651f-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:56.853356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.853264 4525 generic.go:356] "Generic (PLEG): container finished" podID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerID="40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447" exitCode=0 Oct 30 22:36:56.853356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.853321 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerDied","Data":"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447"} Oct 30 22:36:56.853356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.853323 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" Oct 30 22:36:56.853680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.853364 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c" event={"ID":"31b91a93-3d10-4be4-8679-f1aa0252651f","Type":"ContainerDied","Data":"4b999242b2634f7c1d944638e9062e42611a5b67ea8900d6068cbccd3dcac6bc"} Oct 30 22:36:56.853680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.853385 4525 scope.go:122] "RemoveContainer" containerID="40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447" Oct 30 22:36:56.862850 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.862830 4525 scope.go:122] "RemoveContainer" containerID="bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f" Oct 30 22:36:56.870857 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.870833 4525 scope.go:122] "RemoveContainer" containerID="40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447" Oct 30 22:36:56.871142 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:36:56.871124 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447\": container with ID starting with 40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447 not found: ID does not exist" containerID="40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447" Oct 30 22:36:56.871185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.871155 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447"} err="failed to get container status \"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447\": rpc error: code = NotFound desc = could not find container \"40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447\": container with ID starting with 40d2b861d1b76e9f6e2fe0b875798614ef13b893f101cf57159312041fa38447 not found: ID does not exist" Oct 30 22:36:56.871185 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.871174 4525 scope.go:122] "RemoveContainer" containerID="bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f" Oct 30 22:36:56.871408 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:36:56.871393 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f\": container with ID starting with bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f not found: ID does not exist" containerID="bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f" Oct 30 22:36:56.871449 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:56.871413 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f"} err="failed to get container status \"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f\": rpc error: code = NotFound desc = could not find container \"bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f\": container with ID starting with bb7e32b2126c3f4c94da4be30e237ab71b11612be57088af2a579d5c2870599f not found: ID does not exist" Oct 30 22:36:57.018863 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:57.018829 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:36:57.023058 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:57.023035 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-563990-openshift-default-599685759c-k6p5c"] Oct 30 22:36:57.471467 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:36:57.471432 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" path="/var/lib/kubelet/pods/31b91a93-3d10-4be4-8679-f1aa0252651f/volumes" Oct 30 22:37:00.068419 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:37:00.068372 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:37:05.395709 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:37:05.395666 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:37:21.026624 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:37:21.026532 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:37:35.394895 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:37:35.394854 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:37:42.241809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:37:42.241768 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:37:49.673634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:37:49.669750 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:37:49.673634 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:37:49.669890 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:38:05.395959 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:38:05.395919 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:38:13.530598 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:13.530562 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-l7jz6"] Oct 30 22:38:33.793651 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:33.793619 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8ff4cff99-w25dw_a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff/manager/0.log" Oct 30 22:38:35.398731 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:38:35.398697 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:38:35.434809 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:35.434786 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:38:36.514187 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:36.514109 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bsp76_6123e7bb-86b6-417c-9961-0d62abd097b8/proxy/0.log" Oct 30 22:38:37.595891 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:37.595866 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bb5fc9b7-kbvzm_aaf0e822-6aaa-4529-bc45-d62246ca639f/maas-api/1.log" Oct 30 22:38:39.064148 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:39.064117 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bb5fc9b7-kbvzm_aaf0e822-6aaa-4529-bc45-d62246ca639f/maas-api/0.log" Oct 30 22:38:52.024158 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:52.024123 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l7jz6_571e4219-7208-48ea-80dd-f2118d4293d6/limitador/0.log" Oct 30 22:38:53.314847 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:53.314813 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ktljz_93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1/discovery/0.log" Oct 30 22:38:53.692168 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:53.692137 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-2j5st_598b7c39-8df0-4cf2-b736-a2254f9a00e6/kube-auth-proxy/0.log" Oct 30 22:38:54.153473 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:54.153438 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-5xcrz_e5e06632-8ee0-4694-9de1-544e13d57691/istio-proxy/0.log" Oct 30 22:38:55.406006 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:55.405980 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-jlp79_be9bcc39-b9df-4cb4-880a-3fab5a628cb7/istio-proxy/0.log" Oct 30 22:38:55.639414 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:55.639369 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-mndl2_46d47965-7123-48b0-9658-28993d573c42/istio-proxy/0.log" Oct 30 22:38:56.977913 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:56.977871 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6qs7g_ed476948-8416-4669-9229-6cf4011aa55c/payload-pre-processing/0.log" Oct 30 22:38:58.324318 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:58.324261 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8_35cb03d2-e469-4853-a4c1-8ec647eba836/main/0.log" Oct 30 22:38:59.107575 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:59.107536 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-d4t8c_5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa/main/0.log" Oct 30 22:38:59.669995 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:59.669970 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9vnhn_c30d62a7-8f12-41dd-aff1-2bb4d20498e3/main/0.log" Oct 30 22:38:59.956848 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:38:59.956736 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn_71e95dbd-f2d9-47d1-812d-111c027f4ef8/main/0.log" Oct 30 22:39:05.397204 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:39:05.397168 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:39:30.875035 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:30.875000 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-8ff4cff99-w25dw_a84ef14f-a5a5-4b22-b1f5-8bd2c73847ff/manager/0.log" Oct 30 22:39:32.238261 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:32.238222 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:39:33.178641 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:33.178594 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-bsp76_6123e7bb-86b6-417c-9961-0d62abd097b8/proxy/0.log" Oct 30 22:39:34.059312 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:34.059285 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bb5fc9b7-kbvzm_aaf0e822-6aaa-4529-bc45-d62246ca639f/maas-api/1.log" Oct 30 22:39:35.395744 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:39:35.395704 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:39:37.762670 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:37.762638 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l7jz6_571e4219-7208-48ea-80dd-f2118d4293d6/limitador/0.log" Oct 30 22:39:38.876501 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:38.876471 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ktljz_93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1/discovery/0.log" Oct 30 22:39:39.255075 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:39.254996 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-2j5st_598b7c39-8df0-4cf2-b736-a2254f9a00e6/kube-auth-proxy/0.log" Oct 30 22:39:39.716952 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:39.716927 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-5xcrz_e5e06632-8ee0-4694-9de1-544e13d57691/istio-proxy/0.log" Oct 30 22:39:51.098282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:51.098252 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-jlp79_be9bcc39-b9df-4cb4-880a-3fab5a628cb7/istio-proxy/0.log" Oct 30 22:39:51.334332 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:51.334302 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-mndl2_46d47965-7123-48b0-9658-28993d573c42/istio-proxy/0.log" Oct 30 22:39:52.663455 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:52.663426 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6qs7g_ed476948-8416-4669-9229-6cf4011aa55c/payload-pre-processing/0.log" Oct 30 22:39:53.751971 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:53.751942 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-dz7f8_35cb03d2-e469-4853-a4c1-8ec647eba836/main/0.log" Oct 30 22:39:54.560277 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:54.560248 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-d4t8c_5f4aff0e-b9cd-4ae2-a3bf-59dbbd6718aa/main/0.log" Oct 30 22:39:55.070911 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:55.070877 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9vnhn_c30d62a7-8f12-41dd-aff1-2bb4d20498e3/main/0.log" Oct 30 22:39:55.339057 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:39:55.338983 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-sj6qn_71e95dbd-f2d9-47d1-812d-111c027f4ef8/main/0.log" Oct 30 22:40:05.397151 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:40:05.397098 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:40:35.395759 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:40:35.395681 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:40:46.082366 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:40:46.082328 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:40:46.619529 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:40:46.619493 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:41:05.396340 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:41:05.396302 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:41:33.777213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:41:33.777162 4525 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal" podUID="ad6010805d267929c8af4d6a80598dea" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:41:35.398477 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:41:35.398436 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:42:05.396243 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:42:05.396201 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:42:35.396256 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:42:35.396217 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:42:38.248514 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:38.248475 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:42:38.799906 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:38.799870 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685458bf9d-4hg6m_72e7e8d8-b262-47ee-8e76-85f72c893624/manager/0.log" Oct 30 22:42:40.221243 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221207 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mh4rx/must-gather-mbs2t"] Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221566 4525 cpu_manager.go:414] "containerMap: removing container" podUID="633e1166-b903-48e0-9b0b-6454aff7745c" containerName="payload-processing" Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221578 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633e1166-b903-48e0-9b0b-6454aff7745c" containerName="payload-processing" Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221588 4525 cpu_manager.go:414] "containerMap: removing container" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221594 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221624 4525 cpu_manager.go:414] "containerMap: removing container" podUID="387083f2-cd41-4587-bf7c-9989dad40063" containerName="payload-pre-processing" Oct 30 22:42:40.221691 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221629 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387083f2-cd41-4587-bf7c-9989dad40063" containerName="payload-pre-processing" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221698 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="387083f2-cd41-4587-bf7c-9989dad40063" containerName="payload-pre-processing" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221707 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221719 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="633e1166-b903-48e0-9b0b-6454aff7745c" containerName="payload-processing" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221766 4525 cpu_manager.go:414] "containerMap: removing container" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221771 4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.221927 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.221834 4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b91a93-3d10-4be4-8679-f1aa0252651f" containerName="istio-proxy" Oct 30 22:42:40.224882 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.224865 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.277385 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.277352 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh4rx\"/\"openshift-service-ca.crt\"" Oct 30 22:42:40.295002 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.294975 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c59f8982-668e-4a4d-9677-30004b4c7dad-must-gather-output\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.295087 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.295007 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngmq\" (UniqueName: \"kubernetes.io/projected/c59f8982-668e-4a4d-9677-30004b4c7dad-kube-api-access-dngmq\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.327356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.327323 4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mh4rx\"/\"default-dockercfg-2gdt5\"" Oct 30 22:42:40.327356 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.327339 4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh4rx\"/\"kube-root-ca.crt\"" Oct 30 22:42:40.330376 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.330353 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/must-gather-mbs2t"] Oct 30 22:42:40.395785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.395745 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c59f8982-668e-4a4d-9677-30004b4c7dad-must-gather-output\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.395785 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.395787 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngmq\" (UniqueName: \"kubernetes.io/projected/c59f8982-668e-4a4d-9677-30004b4c7dad-kube-api-access-dngmq\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.396108 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.396085 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c59f8982-668e-4a4d-9677-30004b4c7dad-must-gather-output\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.457527 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.457493 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngmq\" (UniqueName: \"kubernetes.io/projected/c59f8982-668e-4a4d-9677-30004b4c7dad-kube-api-access-dngmq\") pod \"must-gather-mbs2t\" (UID: \"c59f8982-668e-4a4d-9677-30004b4c7dad\") " pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.550375 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.550290 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" Oct 30 22:42:40.923702 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.923677 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/must-gather-mbs2t"] Oct 30 22:42:40.925640 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:42:40.925596 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59f8982_668e_4a4d_9677_30004b4c7dad.slice/crio-622313016bbb0eaf1673346f4c1164eaa5146404db6a85fb016420d663944946 WatchSource:0}: Error finding container 622313016bbb0eaf1673346f4c1164eaa5146404db6a85fb016420d663944946: Status 404 returned error can't find the container with id 622313016bbb0eaf1673346f4c1164eaa5146404db6a85fb016420d663944946 Oct 30 22:42:40.927408 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:40.927391 4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:42:41.422219 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:41.422180 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" event={"ID":"c59f8982-668e-4a4d-9677-30004b4c7dad","Type":"ContainerStarted","Data":"622313016bbb0eaf1673346f4c1164eaa5146404db6a85fb016420d663944946"} Oct 30 22:42:42.428837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:42.428748 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" event={"ID":"c59f8982-668e-4a4d-9677-30004b4c7dad","Type":"ContainerStarted","Data":"4a86cdcacfa0c3fc5c30e94645eb0e5fe11cd278f14da3b7b6f45cd8091e528d"} Oct 30 22:42:42.428837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:42.428789 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" event={"ID":"c59f8982-668e-4a4d-9677-30004b4c7dad","Type":"ContainerStarted","Data":"70a593694a074649a28cb6d6e9aa5bd0c39c189d66dfc27c87a585c8286012f0"} Oct 30 22:42:42.546222 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:42:42.546168 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mh4rx/must-gather-mbs2t" podStartSLOduration=1.333192484 podStartE2EDuration="2.546153512s" totalImagesPullingTime="1.212961028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:42.545649198 +0000 UTC m=+4787.219695735" watchObservedRunningTime="2026-10-30 22:42:42.546153512 +0000 UTC m=+4787.220200047" Oct 30 22:43:00.211631 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:00.210771 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q7xnk_f3f8276e-9414-4129-8332-22be9c980723/global-pull-secret-syncer/0.log" Oct 30 22:43:03.680213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:03.680173 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pq7d9_85e468f3-d3a6-4f2d-8964-f18c273b0db3/konnectivity-agent/0.log" Oct 30 22:43:04.074582 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:04.074494 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-53.us-west-1.compute.internal_ad6010805d267929c8af4d6a80598dea/haproxy/0.log" Oct 30 22:43:05.423046 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:43:05.423004 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:43:10.069258 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:10.069172 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-l7jz6_571e4219-7208-48ea-80dd-f2118d4293d6/limitador/0.log" Oct 30 22:43:13.114181 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:13.114144 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/alertmanager/0.log" Oct 30 22:43:13.540579 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:13.540548 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/config-reloader/0.log" Oct 30 22:43:13.965303 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:13.965273 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/kube-rbac-proxy-web/0.log" Oct 30 22:43:14.385666 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:14.385636 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/kube-rbac-proxy/0.log" Oct 30 22:43:14.814452 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:14.814370 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/kube-rbac-proxy-metric/0.log" Oct 30 22:43:15.239650 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:15.239596 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/prom-label-proxy/0.log" Oct 30 22:43:15.661042 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:15.661010 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_db755ab9-3225-430e-b8c0-1fa42bf52a0b/init-config-reloader/0.log" Oct 30 22:43:17.912296 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:17.912260 4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd"] Oct 30 22:43:17.918495 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:17.918467 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:17.974173 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:17.974136 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd"] Oct 30 22:43:18.021746 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.021710 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-podres\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.021940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.021759 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqw8\" (UniqueName: \"kubernetes.io/projected/b7929bc2-c39c-4321-b90a-002983982d28-kube-api-access-zdqw8\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.021940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.021839 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-proc\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.021940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.021892 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-lib-modules\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.021940 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.021950 4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-sys\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.122833 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.122798 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqw8\" (UniqueName: \"kubernetes.io/projected/b7929bc2-c39c-4321-b90a-002983982d28-kube-api-access-zdqw8\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.122838 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-proc\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.122908 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-lib-modules\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.122973 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-sys\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.122980 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-proc\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.123026 4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-podres\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123050 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.123048 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-sys\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.123067 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-lib-modules\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.123313 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.123133 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7929bc2-c39c-4321-b90a-002983982d28-podres\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.185474 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.185399 4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqw8\" (UniqueName: \"kubernetes.io/projected/b7929bc2-c39c-4321-b90a-002983982d28-kube-api-access-zdqw8\") pod \"perf-node-gather-daemonset-jccmd\" (UID: \"b7929bc2-c39c-4321-b90a-002983982d28\") " pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.231426 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.231391 4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:18.606584 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.606557 4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd"] Oct 30 22:43:18.608906 ip-10-0-134-53 kubenswrapper[4525]: W1030 22:43:18.608879 4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7929bc2_c39c_4321_b90a_002983982d28.slice/crio-125ca1a0cdc52300a6f578490abb57d2d20cd5514fdb00d447928e9a572ee9c8 WatchSource:0}: Error finding container 125ca1a0cdc52300a6f578490abb57d2d20cd5514fdb00d447928e9a572ee9c8: Status 404 returned error can't find the container with id 125ca1a0cdc52300a6f578490abb57d2d20cd5514fdb00d447928e9a572ee9c8 Oct 30 22:43:18.616759 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.616734 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" event={"ID":"b7929bc2-c39c-4321-b90a-002983982d28","Type":"ContainerStarted","Data":"125ca1a0cdc52300a6f578490abb57d2d20cd5514fdb00d447928e9a572ee9c8"} Oct 30 22:43:18.811152 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:18.811094 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gnht_bc1fc34a-ac92-4787-9859-2535ff564f73/node-exporter/0.log" Oct 30 22:43:19.231029 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:19.231003 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gnht_bc1fc34a-ac92-4787-9859-2535ff564f73/kube-rbac-proxy/0.log" Oct 30 22:43:19.623585 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:19.623544 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" event={"ID":"b7929bc2-c39c-4321-b90a-002983982d28","Type":"ContainerStarted","Data":"02f0906112e0e5651c6e510cbd18f15db65168af84afe35a0024192397bccb1b"} Oct 30 22:43:19.623808 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:19.623659 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:19.653521 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:19.653494 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gnht_bc1fc34a-ac92-4787-9859-2535ff564f73/init-textfile/0.log" Oct 30 22:43:19.741260 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:19.741209 4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" podStartSLOduration=2.741195952 podStartE2EDuration="2.741195952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:19.740445053 +0000 UTC m=+4824.414491592" watchObservedRunningTime="2026-10-30 22:43:19.741195952 +0000 UTC m=+4824.415242484" Oct 30 22:43:25.640090 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:25.640061 4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mh4rx/perf-node-gather-daemonset-jccmd" Oct 30 22:43:35.396503 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:43:35.396459 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:43:39.941436 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:39.941402 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66f8777668-2kc55_b4d86910-b2d4-4888-8f18-f07569d96377/console/0.log" Oct 30 22:43:43.692879 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.692846 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:43:43.693719 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.693684 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" containerName="istio-proxy" containerID="cri-o://9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932" gracePeriod=30 Oct 30 22:43:43.742256 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.742211 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:43:43.742472 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.742435 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" podUID="46d47965-7123-48b0-9658-28993d573c42" containerName="istio-proxy" containerID="cri-o://898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9" gracePeriod=30 Oct 30 22:43:43.743904 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.743874 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 22:43:43.744067 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:43.744048 4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" containerID="cri-o://dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a" gracePeriod=30 Oct 30 22:43:48.623669 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:48.623599 4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jlp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:43:48.624183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:48.623712 4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:43:49.005104 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.005067 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:43:49.041230 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.041210 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:43:49.044198 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.044179 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 22:43:49.142556 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142528 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142571 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142644 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142666 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142692 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142720 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142751 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.142780 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142780 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142826 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142855 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142890 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142869 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142916 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142952 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142965 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.142982 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143017 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143055 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143083 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143132 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143132 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.143183 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143165 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143204 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143229 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143289 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143340 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143379 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143429 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143455 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143470 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\") pod \"e5e06632-8ee0-4694-9de1-544e13d57691\" (UID: \"e5e06632-8ee0-4694-9de1-544e13d57691\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143502 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\") pod \"46d47965-7123-48b0-9658-28993d573c42\" (UID: \"46d47965-7123-48b0-9658-28993d573c42\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143539 4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\") pod \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\" (UID: \"be9bcc39-b9df-4cb4-880a-3fab5a628cb7\") " Oct 30 22:43:49.143930 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143924 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143946 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d47965-7123-48b0-9658-28993d573c42-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143968 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143982 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.143998 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144011 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144410 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.144543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144417 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144580 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144591 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144679 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145128 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.144685 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145394 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.145211 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145394 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.145364 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.145535 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.145499 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.146177 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.146110 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.146905 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.146877 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "kube-api-access-vhg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.147371 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147251 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:49.147371 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147324 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:49.147543 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147376 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.147871 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147843 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.147970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147884 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "kube-api-access-lndvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.147970 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.147928 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.148109 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.148094 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo" pod "46d47965-7123-48b0-9658-28993d573c42" (UID: "46d47965-7123-48b0-9658-28993d573c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:49.148166 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.148113 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb" pod "e5e06632-8ee0-4694-9de1-544e13d57691" (UID: "e5e06632-8ee0-4694-9de1-544e13d57691"). InnerVolumeSpecName "kube-api-access-8jtkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.148538 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.148520 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:49.148944 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.148928 4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token" pod "be9bcc39-b9df-4cb4-880a-3fab5a628cb7" (UID: "be9bcc39-b9df-4cb4-880a-3fab5a628cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244851 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244878 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhg4s\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-kube-api-access-vhg4s\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244889 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d47965-7123-48b0-9658-28993d573c42-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244897 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lndvw\" (UniqueName: \"kubernetes.io/projected/46d47965-7123-48b0-9658-28993d573c42-kube-api-access-lndvw\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244906 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e06632-8ee0-4694-9de1-544e13d57691-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244919 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.244929 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244931 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244946 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244959 4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-workload-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244973 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.244990 4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-podinfo\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245005 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245018 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d47965-7123-48b0-9658-28993d573c42-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245032 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245044 4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-token\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245059 4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-data\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245073 4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtkb\" (UniqueName: \"kubernetes.io/projected/e5e06632-8ee0-4694-9de1-544e13d57691-kube-api-access-8jtkb\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245087 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245099 4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e06632-8ee0-4694-9de1-544e13d57691-workload-certs\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245112 4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-istio-envoy\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245125 4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9bcc39-b9df-4cb4-880a-3fab5a628cb7-credential-socket\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.245282 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.245138 4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e06632-8ee0-4694-9de1-544e13d57691-istiod-ca-cert\") on node \"ip-10-0-134-53.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.511680 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.511582 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bblgp_98b136c6-693a-44bf-b141-b179fbbdc441/dns/0.log" Oct 30 22:43:49.780837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.780756 4525 generic.go:356] "Generic (PLEG): container finished" podID="e5e06632-8ee0-4694-9de1-544e13d57691" containerID="9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932" exitCode=0 Oct 30 22:43:49.780837 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.780828 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" Oct 30 22:43:49.781342 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.780834 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" event={"ID":"e5e06632-8ee0-4694-9de1-544e13d57691","Type":"ContainerDied","Data":"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932"} Oct 30 22:43:49.781342 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.780921 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz" event={"ID":"e5e06632-8ee0-4694-9de1-544e13d57691","Type":"ContainerDied","Data":"0914691adb90cea4e221dd7bc812143b8c90375ca143cf446ec5a66c50c6ddea"} Oct 30 22:43:49.781342 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.780954 4525 scope.go:122] "RemoveContainer" containerID="9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932" Oct 30 22:43:49.782294 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.782269 4525 generic.go:356] "Generic (PLEG): container finished" podID="46d47965-7123-48b0-9658-28993d573c42" containerID="898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9" exitCode=0 Oct 30 22:43:49.782421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.782337 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" Oct 30 22:43:49.782421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.782358 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" event={"ID":"46d47965-7123-48b0-9658-28993d573c42","Type":"ContainerDied","Data":"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9"} Oct 30 22:43:49.782421 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.782417 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2" event={"ID":"46d47965-7123-48b0-9658-28993d573c42","Type":"ContainerDied","Data":"4ac3b32e0b47171f6cee5141f63f9ccd496adac6b4b8a134032860d112b32eb3"} Oct 30 22:43:49.784095 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.784069 4525 generic.go:356] "Generic (PLEG): container finished" podID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" containerID="dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a" exitCode=0 Oct 30 22:43:49.784213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.784104 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" event={"ID":"be9bcc39-b9df-4cb4-880a-3fab5a628cb7","Type":"ContainerDied","Data":"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a"} Oct 30 22:43:49.784213 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.784141 4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" event={"ID":"be9bcc39-b9df-4cb4-880a-3fab5a628cb7","Type":"ContainerDied","Data":"04029fb5d38d5a3d0ef6e05fa85475429804fe27463438ed0fbef3c7a6ab1b23"} Oct 30 22:43:49.784296 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.784217 4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79" Oct 30 22:43:49.790661 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.790646 4525 scope.go:122] "RemoveContainer" containerID="9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932" Oct 30 22:43:49.790910 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:43:49.790887 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932\": container with ID starting with 9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932 not found: ID does not exist" containerID="9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932" Oct 30 22:43:49.790958 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.790921 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932"} err="failed to get container status \"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932\": rpc error: code = NotFound desc = could not find container \"9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932\": container with ID starting with 9390d44fe4672de5c575394a5d9acbbca5a49c79cddf8fde55acb94b1dfba932 not found: ID does not exist" Oct 30 22:43:49.791009 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.790961 4525 scope.go:122] "RemoveContainer" containerID="898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9" Oct 30 22:43:49.802640 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.802593 4525 scope.go:122] "RemoveContainer" containerID="898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9" Oct 30 22:43:49.802942 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:43:49.802920 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9\": container with ID starting with 898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9 not found: ID does not exist" containerID="898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9" Oct 30 22:43:49.802994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.802953 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9"} err="failed to get container status \"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9\": rpc error: code = NotFound desc = could not find container \"898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9\": container with ID starting with 898d4478323cf7f4772e1fac1566aeba2bb194dd0ec326dd781a22bd24f229f9 not found: ID does not exist" Oct 30 22:43:49.802994 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.802970 4525 scope.go:122] "RemoveContainer" containerID="dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a" Oct 30 22:43:49.811226 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.811210 4525 scope.go:122] "RemoveContainer" containerID="dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a" Oct 30 22:43:49.811544 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:43:49.811526 4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a\": container with ID starting with dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a not found: ID does not exist" containerID="dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a" Oct 30 22:43:49.811629 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.811555 4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a"} err="failed to get container status \"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a\": rpc error: code = NotFound desc = could not find container \"dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a\": container with ID starting with dfa26d9dcd7551c4e2f0ffdf79176d2ea562af7a143a0c2fa214d4079f88173a not found: ID does not exist" Oct 30 22:43:49.933739 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.933709 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bblgp_98b136c6-693a-44bf-b141-b179fbbdc441/kube-rbac-proxy/0.log" Oct 30 22:43:49.965153 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:49.965120 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 22:43:50.014658 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:50.014620 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jlp79"] Oct 30 22:43:50.084630 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:50.084531 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:43:50.088629 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:50.088577 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mndl2"] Oct 30 22:43:50.198781 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:50.198748 4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:43:50.203699 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:50.203674 4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-5xcrz"] Oct 30 22:43:51.472454 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:51.472418 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d47965-7123-48b0-9658-28993d573c42" path="/var/lib/kubelet/pods/46d47965-7123-48b0-9658-28993d573c42/volumes" Oct 30 22:43:51.473065 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:51.473043 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9bcc39-b9df-4cb4-880a-3fab5a628cb7" path="/var/lib/kubelet/pods/be9bcc39-b9df-4cb4-880a-3fab5a628cb7/volumes" Oct 30 22:43:51.473507 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:51.473488 4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e06632-8ee0-4694-9de1-544e13d57691" path="/var/lib/kubelet/pods/e5e06632-8ee0-4694-9de1-544e13d57691/volumes" Oct 30 22:43:52.060143 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:52.060110 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-czp4b_9a817f0e-4473-409d-9638-459e8b0a066f/dns-node-resolver/0.log" Oct 30 22:43:54.908583 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:54.908547 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6fddbbf884-7gkgc_0c75964e-1c00-43ce-ad8c-e3e5010f5d29/registry/0.log" Oct 30 22:43:56.200814 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:43:56.200789 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pgkp4_975bc6cc-1f63-4ae5-ac26-771225c652b4/node-ca/0.log" Oct 30 22:44:01.441941 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:44:01.441903 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ktljz_93b1522f-f6a1-4cb1-9a10-187bbc7e2ca1/discovery/0.log" Oct 30 22:44:03.084268 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:44:03.084238 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcd447999-2j5st_598b7c39-8df0-4cf2-b736-a2254f9a00e6/kube-auth-proxy/0.log" Oct 30 22:44:04.779449 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:44:04.779416 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6qs7g_ed476948-8416-4669-9229-6cf4011aa55c/payload-pre-processing/0.log" Oct 30 22:44:05.395818 ip-10-0-134-53 kubenswrapper[4525]: E1030 22:44:05.395780 4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wsk2z" containerName="extractor" Oct 30 22:44:09.312571 ip-10-0-134-53 kubenswrapper[4525]: I1030 22:44:09.312526 4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ckh6t_5686b209-4171-4cc7-8f73-b128ca2d068e/serve-healthcheck-canary/0.log"